Nov 03 22:04:23.344467 ip-10-0-134-57 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 22:04:23.344477 ip-10-0-134-57 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 22:04:23.344484 ip-10-0-134-57 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 22:04:23.344735 ip-10-0-134-57 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 22:04:34.590675 ip-10-0-134-57 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 22:04:34.590693 ip-10-0-134-57 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 27453300eb664d36a070ab89f7d83c8d -- Nov 03 22:06:46.809780 ip-10-0-134-57 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:06:47.195610 ip-10-0-134-57 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:06:47.195610 ip-10-0-134-57 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:06:47.195610 ip-10-0-134-57 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:06:47.195610 ip-10-0-134-57 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:06:47.198662 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198575 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:47.198662 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198661 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:47.198662 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198664 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:47.198662 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198668 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198671 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198674 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198677 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198681 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198683 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198686 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198689 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198692 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198694 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198697 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198700 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198703 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198705 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198708 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198710 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198713 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198717 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198721 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:47.198782 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198724 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198727 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198729 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198732 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198734 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198737 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198740 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198749 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198752 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198755 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198758 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198760 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198763 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198766 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198769 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198771 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198774 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198777 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198780 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198783 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:47.199224 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198786 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198789 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198792 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198794 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198797 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198800 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198804 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198806 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198809 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198811 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198814 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198817 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198819 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198822 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198825 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198828 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198830 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198833 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198836 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198838 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:47.199718 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198841 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198843 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198846 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198849 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198851 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198854 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198857 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198860 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198862 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198865 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198867 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198870 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198872 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198875 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198878 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198880 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198883 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198886 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198889 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198892 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:47.200228 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198894 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198897 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198900 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198902 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198905 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198908 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198910 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198913 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198915 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198918 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198922 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198925 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198928 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198931 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198934 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198936 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198939 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198942 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198946 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:47.200731 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198949 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198951 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198954 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198956 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.198959 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199764 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199771 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199774 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199777 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199780 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199783 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199786 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199789 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199792 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199794 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199797 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199800 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199802 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199805 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199808 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:47.201180 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199811 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199813 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199816 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199819 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199821 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199824 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199827 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199829 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199832 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199835 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199838 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199840 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199844 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199849 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199852 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199855 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199858 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199861 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199864 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:47.201680 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199866 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199869 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199874 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199877 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199880 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199883 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199886 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199889 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199892 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199895 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199897 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199900 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199902 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199905 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199908 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199910 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199913 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199915 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199918 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:47.202137 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199921 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199924 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199927 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199930 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199932 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199935 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199938 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199940 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199943 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199945 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199948 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199951 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199953 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199956 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199958 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199962 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199965 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199967 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199970 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:47.202593 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199972 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199976 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199978 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199981 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199983 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199986 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199988 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199991 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199993 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199996 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.199998 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200001 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200003 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200006 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200008 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200011 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200014 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200017 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200019 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200022 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:47.203095 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200024 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200027 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200030 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200032 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200035 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200037 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200040 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200042 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200046 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200049 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200052 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200054 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200058 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200062 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200572 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200578 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200581 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200585 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200588 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:47.203576 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200591 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200594 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200598 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200601 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200605 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200608 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200610 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200615 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200618 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200621 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200623 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200627 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200629 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200632 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200634 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200637 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200640 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200643 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200645 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200648 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:47.204023 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200650 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200653 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200656 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200659 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200661 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200664 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200666 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200669 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200672 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200674 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200677 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200679 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200682 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200684 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200687 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200689 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200692 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200694 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200696 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200701 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:47.204527 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200703 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200706 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200708 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200710 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200713 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200715 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200718 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200721 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200724 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200726 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200747 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200751 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200755 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200758 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200761 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200764 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200767 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200770 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200773 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200775 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:47.205016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200778 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200781 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200784 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200787 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200789 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200792 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200794 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200797 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200799 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200803 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200807 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200810 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200813 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200816 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200818 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200821 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200823 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200826 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200829 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:47.205481 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200831 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200834 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200836 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200839 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200841 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200844 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200846 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200849 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200852 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200854 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200857 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200859 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200862 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200864 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200867 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200870 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200872 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200875 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200877 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200881 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:47.205947 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200884 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.200886 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.200974 2585 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.200982 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.200988 2585 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.200994 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.200999 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201002 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201006 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201011 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201014 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201017 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201021 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201024 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201027 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201030 2585 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201033 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201036 2585 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201039 2585 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201042 2585 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201047 2585 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201050 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201053 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201056 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:06:47.206418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201062 2585 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201066 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201069 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201073 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201076 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201080 2585 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201083 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201086 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201089 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201092 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201097 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201100 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201103 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201106 2585 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201109 2585 flags.go:64] FLAG: --enable-server="true" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201112 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201117 2585 flags.go:64] FLAG: --event-burst="100" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201120 2585 flags.go:64] FLAG: --event-qps="50" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201123 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201127 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201130 2585 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201134 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201138 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201141 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201144 2585 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:06:47.206976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201147 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201150 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201153 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201156 2585 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201159 2585 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201162 2585 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201165 2585 flags.go:64] FLAG: --feature-gates="" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201170 2585 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201173 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201177 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201180 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201184 2585 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201187 2585 flags.go:64] FLAG: --help="false" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201191 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201194 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201197 2585 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201200 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201203 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201207 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201210 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201213 2585 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201216 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201219 2585 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:06:47.207733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201222 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201225 2585 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201228 2585 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201231 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201234 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201237 2585 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201240 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201243 2585 flags.go:64] FLAG: --lock-file="" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201246 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201248 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201251 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201257 2585 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201260 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201263 2585 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201266 2585 flags.go:64] FLAG: --logging-format="text" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201269 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201275 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201278 2585 flags.go:64] FLAG: --manifest-url="" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201281 2585 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201285 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201288 2585 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201292 2585 flags.go:64] FLAG: --max-pods="110" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201296 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201299 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201302 2585 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:06:47.208285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201305 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201310 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201313 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201316 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201324 2585 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201327 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201331 2585 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201334 2585 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201337 2585 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201340 2585 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201343 2585 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201346 2585 flags.go:64] FLAG: --port="10250" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201349 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201352 2585 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-005dede14741bddc3" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201355 2585 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201358 2585 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201361 2585 flags.go:64] FLAG: --register-node="true" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201364 2585 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201368 2585 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201371 2585 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201373 2585 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201376 2585 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201381 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201385 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201389 2585 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201392 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:06:47.208904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201395 2585 flags.go:64] FLAG: --runonce="false" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201398 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201401 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201404 2585 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201407 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201410 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201414 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201418 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201421 2585 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201425 2585 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201428 2585 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201431 2585 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201434 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201438 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201441 2585 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201444 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201449 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201452 2585 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201455 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201459 2585 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201462 2585 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201465 2585 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201468 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201471 2585 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201474 2585 flags.go:64] FLAG: --v="2" Nov 03 22:06:47.209593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201481 2585 flags.go:64] FLAG: --version="false" Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201486 2585 flags.go:64] FLAG: --vmodule="" Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201490 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201508 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201647 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201651 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201654 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201657 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201661 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201665 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201667 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201671 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201673 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201676 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201681 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201684 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201686 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201689 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201692 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201695 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:47.210192 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201697 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201700 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201703 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201706 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201708 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201711 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201713 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201716 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201718 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201721 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201723 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201726 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201728 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201731 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201733 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201736 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201740 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201743 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201746 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201749 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:47.210681 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201751 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201754 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201757 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201760 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201763 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201765 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201770 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201772 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201775 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201777 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201780 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201782 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201785 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201787 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201790 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201792 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201795 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201797 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201800 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:47.211164 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201802 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201805 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201807 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201810 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201813 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201815 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201818 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201820 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201823 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201826 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201829 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201832 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201834 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201837 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201840 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201842 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201845 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201847 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201850 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201855 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:47.211622 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201857 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201861 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201865 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201868 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201870 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201873 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201876 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201878 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201881 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201883 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201886 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201889 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201891 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201894 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201896 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201899 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201901 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201904 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201907 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:47.212130 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201909 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201911 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201915 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201918 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201921 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201923 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201926 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201928 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201931 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201933 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201936 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.201939 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.201966 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202002 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202006 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202010 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:47.212601 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202012 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202016 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202018 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202021 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202024 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202027 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202030 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202032 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202035 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202038 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202040 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202043 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202045 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202048 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202050 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202053 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202055 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202058 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202061 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:47.212992 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202065 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202067 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202070 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202072 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202075 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202077 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202080 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202083 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202085 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202090 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202094 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202097 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202099 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202102 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202105 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202107 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202110 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202112 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202115 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202118 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:47.213442 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202120 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202123 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202125 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202128 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202131 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202136 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202140 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202143 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202146 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202149 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202152 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202154 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202159 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202162 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202164 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202167 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202170 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202172 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202175 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:47.213934 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202177 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202180 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202184 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202186 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202190 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202192 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202195 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202198 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202201 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202203 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202205 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202208 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202210 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202213 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202215 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202218 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202220 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202223 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202226 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202230 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:47.214388 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202234 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202237 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202240 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202244 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202247 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202251 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202253 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202256 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202258 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202261 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202264 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202266 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202269 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202271 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202276 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202278 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202281 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202284 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202287 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202290 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:47.214933 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202292 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202295 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202297 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202300 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.202302 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.209313 2585 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.209329 2585 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.209346 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.209351 2585 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209437 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209442 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209445 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209448 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209451 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209454 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209457 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209461 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209465 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209469 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:47.215401 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209472 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209474 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209478 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209482 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209485 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209488 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209490 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209508 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209511 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209514 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209523 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209527 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209529 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209532 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209535 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209537 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209540 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209542 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209545 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:47.215864 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209548 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209551 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209553 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209556 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209559 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209561 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209564 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209567 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209570 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209572 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209575 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209577 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209580 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209583 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209585 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209588 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209590 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209593 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209596 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:47.216541 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209598 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209601 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209604 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209606 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209609 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209617 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209620 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209623 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209626 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209628 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209631 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209633 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209636 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209639 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209642 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209644 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209647 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209650 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209653 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209655 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:47.217272 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209658 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209661 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209663 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209665 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209668 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209670 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209673 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209675 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209678 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209680 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209683 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209685 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209688 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209690 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209693 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209696 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209698 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209701 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209710 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:47.217954 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209713 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209716 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209718 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209721 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209723 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209726 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209729 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209732 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209734 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209737 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209739 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209742 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209745 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209747 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209750 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209752 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209755 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209758 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:47.218429 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209760 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.209788 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209976 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209980 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209983 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209986 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209989 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209991 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209994 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209996 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.209999 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210002 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210004 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210007 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210016 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210020 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:47.218943 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210022 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210025 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210028 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210031 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210033 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210036 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210038 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210041 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210044 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210046 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210049 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210051 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210054 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210057 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210059 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210062 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210064 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210067 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210069 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:47.219333 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210072 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210075 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210079 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210082 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210084 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210087 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210090 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210092 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210094 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210097 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210100 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210102 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210110 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210113 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210116 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210119 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210122 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210124 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210127 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210130 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:47.219941 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210132 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210135 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210137 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210140 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210142 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210145 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210148 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210150 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210153 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210155 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210158 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210160 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210163 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210165 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210168 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210171 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210173 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210176 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210178 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210181 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:47.220462 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210183 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210185 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210188 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210191 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210194 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210202 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210206 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210209 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210213 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210217 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210220 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210223 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210225 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210229 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210231 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210234 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210236 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210239 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210241 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:47.221016 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210244 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210246 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210249 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210252 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210254 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210256 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210259 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210261 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210264 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210266 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210269 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210271 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210274 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210276 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.210299 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:06:47.221474 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210346 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210351 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210355 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210358 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210367 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210370 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210372 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210375 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210378 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210380 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210383 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210386 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210388 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210391 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210393 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210396 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210398 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210403 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210407 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:47.222011 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210410 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210413 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210416 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210418 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210421 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210423 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210426 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210428 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210430 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210433 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210436 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210438 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210440 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210443 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210445 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210448 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210451 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210453 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210462 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:47.222468 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210465 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210468 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210470 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210473 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210475 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210478 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210481 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210483 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210486 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210488 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210504 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210509 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210511 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210515 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210518 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210520 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210523 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210525 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210528 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:47.222921 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210530 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210533 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210535 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210538 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210541 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210543 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210546 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210548 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210551 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210553 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210556 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210559 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210561 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210570 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210572 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210575 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210577 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210580 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210583 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:47.223410 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210586 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210590 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210593 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210595 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210598 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210601 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210604 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210607 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210609 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210612 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210614 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210617 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210619 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210622 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210624 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210626 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210629 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210631 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210634 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210637 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210639 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:47.224120 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210642 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210647 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210650 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210654 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210656 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210658 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210667 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210669 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:47.210672 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.210733 2585 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.212655 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.213543 2585 server.go:1014] "Starting client certificate rotation" Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.213636 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.213678 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.216711 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:06:47.224763 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.218666 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:06:47.228092 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.228074 2585 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:06:47.231183 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.231167 2585 log.go:25] "Validated CRI v1 image API" Nov 03 22:06:47.231930 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.231896 2585 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:06:47.235168 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.235145 2585 fs.go:135] Filesystem UUIDs: map[0a876746-fc48-41f1-a8eb-5503b5d72f4c:/dev/nvme0n1p3 52e0cd8e-3ee6-403f-9500-c211b0842cda:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 22:06:47.235238 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.235165 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 22:06:47.240330 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.240216 2585 manager.go:217] Machine: {Timestamp:2026-11-03 22:06:47.239010281 +0000 UTC m=+0.318459308 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100931 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa114374c4bc49556fa5e80d5a785 SystemUUID:ec2aa114-374c-4bc4-9556-fa5e80d5a785 BootID:27453300-eb66-4d36-a070-ab89f7d83c8d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:cc:fd:92:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:cc:fd:92:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:34:c9:8a:83:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:06:47.240330 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.240329 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:06:47.240460 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.240448 2585 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:06:47.241686 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.241660 2585 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:06:47.241819 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.241689 2585 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-57.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:06:47.241859 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.241821 2585 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:06:47.241859 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.241829 2585 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:06:47.241915 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.241874 2585 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:06:47.241915 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.241892 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:06:47.242532 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.242521 2585 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:06:47.246537 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.246525 2585 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:06:47.246649 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.246639 2585 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:06:47.247474 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.247465 2585 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:06:47.247523 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.247478 2585 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:06:47.247523 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.247489 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:06:47.247523 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.247508 2585 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:06:47.247523 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.247515 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:06:47.248021 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.248006 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:06:47.251585 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.250388 2585 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:06:47.254894 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.254875 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:06:47.255700 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.255684 2585 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:06:47.255786 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.255714 2585 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:06:47.255881 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.255869 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:06:47.255929 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.255887 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:06:47.255929 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.255893 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:06:47.255929 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.255899 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:06:47.255929 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.255905 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:06:47.255929 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.255910 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:06:47.256299 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.256290 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:06:47.256299 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.256298 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:06:47.256363 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.256306 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:06:47.256363 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.256312 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:06:47.256363 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.256320 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:06:47.256363 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.256329 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:06:47.257300 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.257290 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:06:47.257300 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.257300 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:06:47.261017 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.261003 2585 server.go:1263] "Started kubelet" Nov 03 22:06:47.261158 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.261108 2585 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:06:47.261195 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.261141 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:06:47.261222 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.261196 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:06:47.261788 ip-10-0-134-57 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:06:47.262409 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.262393 2585 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:06:47.262956 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.262941 2585 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:06:47.266978 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.266958 2585 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 22:06:47.269914 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.269892 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:06:47.270857 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.270838 2585 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:06:47.270934 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.270918 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:47.271347 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.271329 2585 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:06:47.271421 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.271359 2585 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:06:47.271421 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.271327 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:06:47.271521 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.271458 2585 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:06:47.271521 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.271466 2585 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:06:47.271817 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.271801 2585 factory.go:55] Registering systemd factory Nov 03 22:06:47.271817 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.271818 2585 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:06:47.272042 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.272028 2585 factory.go:153] Registering CRI-O factory Nov 03 22:06:47.272042 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.272039 2585 factory.go:223] Registration of the crio container factory successfully Nov 03 22:06:47.272181 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.272138 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:06:47.272237 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.272182 2585 factory.go:103] Registering Raw factory Nov 03 22:06:47.272237 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.272193 2585 manager.go:1199] Started watching for new ooms in manager Nov 03 22:06:47.273399 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.273382 2585 manager.go:319] Starting recovery of all containers Nov 03 22:06:47.285770 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.285748 2585 manager.go:324] Recovery completed Nov 03 22:06:47.290204 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.290188 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:47.292117 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.292089 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:47.292199 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.292134 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:47.292199 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.292152 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:47.292779 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.292764 2585 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:06:47.292779 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.292776 2585 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:06:47.292911 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.292797 2585 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:06:47.292911 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.292766 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:47.293534 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.293491 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:47.293596 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.293555 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:47.293596 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.293572 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:47.371907 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.371882 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:47.394261 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.394233 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-57.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 22:06:47.394261 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.394237 2585 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-57.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 22:06:47.395297 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.394256 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-57.us-west-2.compute.internal.18e4a84ea0eaff86 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-57.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-57.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:06:47.260970886 +0000 UTC m=+0.340419912,LastTimestamp:2026-11-03 22:06:47.260970886 +0000 UTC m=+0.340419912,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-57.us-west-2.compute.internal,}" Nov 03 22:06:47.432325 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.432306 2585 policy_none.go:50] "Start" Nov 03 22:06:47.432325 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.432328 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:06:47.432444 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.432346 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:06:47.434316 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.434306 2585 policy_none.go:44] "Start" Nov 03 22:06:47.454458 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.454361 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-57.us-west-2.compute.internal.18e4a84ea2c63c5e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-57.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-57.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-57.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:06:47.292116062 +0000 UTC m=+0.371565099,LastTimestamp:2026-11-03 22:06:47.292116062 +0000 UTC m=+0.371565099,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-57.us-west-2.compute.internal,}" Nov 03 22:06:47.454458 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.454448 2585 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 22:06:47.454581 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.454465 2585 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-57.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 22:06:47.454581 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.454465 2585 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 22:06:47.459057 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.459023 2585 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:06:47.461453 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.461441 2585 manager.go:341] "Starting Device Plugin manager" Nov 03 22:06:47.461533 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.461517 2585 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:06:47.461533 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.461529 2585 server.go:87] "Starting device plugin registration server" Nov 03 22:06:47.461735 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.461725 2585 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:06:47.461785 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.461736 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:06:47.461861 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.461847 2585 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:06:47.461936 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.461924 2585 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:06:47.461936 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.461935 2585 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:06:47.462488 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.462471 2585 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:06:47.462562 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.462534 2585 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:47.525168 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.525146 2585 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:06:47.525733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.525195 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t6tzd" Nov 03 22:06:47.551263 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.551191 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-57.us-west-2.compute.internal.18e4a84ea2c69d38 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-57.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-57.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-57.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:06:47.292140856 +0000 UTC m=+0.371589885,LastTimestamp:2026-11-03 22:06:47.292140856 +0000 UTC m=+0.371589885,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-57.us-west-2.compute.internal,}" Nov 03 22:06:47.562301 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.562282 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:47.563117 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.563100 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:47.563209 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.563147 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:47.563209 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.563164 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:47.563209 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.563200 2585 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.564125 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.564103 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:06:47.565521 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.565485 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:06:47.565584 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.565528 2585 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:06:47.565584 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.565549 2585 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:06:47.565689 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.565586 2585 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 22:06:47.619948 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.619931 2585 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:47.620443 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.620429 2585 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.620567 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.620452 2585 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-57.us-west-2.compute.internal\": node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:47.666365 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.666324 2585 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal"] Nov 03 22:06:47.666457 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.666397 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:47.668205 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.668192 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:47.668266 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.668219 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:47.668266 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.668236 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:47.670453 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.670442 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:47.670627 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.670610 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.670663 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.670643 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:47.671232 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.671213 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:47.671320 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.671246 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:47.671320 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.671258 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:47.671385 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.671332 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:47.671385 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.671357 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:47.671385 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.671368 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:47.673557 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.673544 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.673627 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.673568 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:47.673849 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.673829 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.673898 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.673864 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56a5599a8af5a3ab98eedd947c314121-config\") pod \"kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"56a5599a8af5a3ab98eedd947c314121\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.673898 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.673892 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.674279 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.674262 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:47.674336 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.674284 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:47.674336 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.674298 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:47.700937 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.700920 2585 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" node="ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.703818 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.703804 2585 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" node="ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.743248 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.743227 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:47.774655 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.774625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.774748 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.774661 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.774748 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.774681 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56a5599a8af5a3ab98eedd947c314121-config\") pod \"kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"56a5599a8af5a3ab98eedd947c314121\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.774748 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.774734 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56a5599a8af5a3ab98eedd947c314121-config\") pod \"kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"56a5599a8af5a3ab98eedd947c314121\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.774842 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.774744 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.774842 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:47.774744 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:47.843903 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.843878 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:47.944553 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:47.944526 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:48.004855 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.004801 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:48.006956 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.006938 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:48.045092 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:48.044970 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:48.146056 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:48.146026 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:48.215397 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.215358 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 22:06:48.215984 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.215580 2585 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:06:48.215984 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:48.215620 2585 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3890ba9e34074c9bb3a6df7076aca19-87013095a60c6b5c.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-57.us-west-2.compute.internal.18e4a84ea2c63c5e\": read tcp 10.0.134.57:54824->18.217.15.206:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-57.us-west-2.compute.internal.18e4a84ea2c63c5e default 5389 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-57.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-57.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-57.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:06:47 +0000 UTC,LastTimestamp:2026-11-03 22:06:47.668204738 +0000 UTC m=+0.747653765,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-57.us-west-2.compute.internal,}" Nov 03 22:06:48.246813 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:48.246774 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:48.347056 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:48.346974 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:48.446855 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.446829 2585 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:48.447932 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:48.447913 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:48.531120 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.531080 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:01:48 +0000 UTC" deadline="2028-07-30 15:59:38.206683222 +0000 UTC" Nov 03 22:06:48.531120 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.531115 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15233h52m49.675572852s" Nov 03 22:06:48.548125 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:48.548098 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:48.556402 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.556380 2585 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:48.590188 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.590153 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:48.591022 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.591006 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:48.591092 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.591037 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:48.591092 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.591048 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:48.649155 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:48.649103 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:06:48.675549 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.675527 2585 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:48.772031 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.772005 2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:48.837038 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.837012 2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:06:48.838035 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.838021 2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:06:48.900207 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:48.900147 2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:06:49.230435 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.230395 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad3247384fc65111dcedea645af8378.slice/crio-6bc001e10574ea0052e18de1dcac1a0610084d34cee3c7266a718828f1b0fcda WatchSource:0}: Error finding container 6bc001e10574ea0052e18de1dcac1a0610084d34cee3c7266a718828f1b0fcda: Status 404 returned error can't find the container with id 6bc001e10574ea0052e18de1dcac1a0610084d34cee3c7266a718828f1b0fcda Nov 03 22:06:49.237335 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.237314 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:06:49.240654 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.240629 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a5599a8af5a3ab98eedd947c314121.slice/crio-0d0c414cba1b5256629dc7cd6b00f18c908e6f4e6d3ffedd9791bcf723e1df8d WatchSource:0}: Error finding container 0d0c414cba1b5256629dc7cd6b00f18c908e6f4e6d3ffedd9791bcf723e1df8d: Status 404 returned error can't find the container with id 0d0c414cba1b5256629dc7cd6b00f18c908e6f4e6d3ffedd9791bcf723e1df8d Nov 03 22:06:49.248122 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.248107 2585 apiserver.go:52] "Watching apiserver" Nov 03 22:06:49.271484 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.271460 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 22:06:49.364360 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.364333 2585 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:06:49.364732 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.364710 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-x78ps","kube-system/global-pull-secret-syncer-lwk4s","openshift-dns/node-resolver-d8jgp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-9bqcf","kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-4s4p6","openshift-image-registry/node-ca-v6n4q","openshift-multus/multus-additional-cni-plugins-v7c9n","openshift-multus/network-metrics-daemon-xkldp","openshift-ovn-kubernetes/ovnkube-node-8zltr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw","openshift-multus/multus-796dw","openshift-network-operator/iptables-alerter-2swns"] Nov 03 22:06:49.368549 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.368529 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:49.368632 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.368614 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:06:49.370780 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.370759 2585 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:49.370892 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.370777 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-796dw" Nov 03 22:06:49.374370 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.374346 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.376660 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.376638 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.378863 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.378847 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:49.378971 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.378947 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:06:49.381224 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.381209 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.383387 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.383372 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.385754 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385654 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.385754 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385740 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-etc-kubernetes\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.385879 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385776 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-netns\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.385879 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-multus\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.385879 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385832 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-conf-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.385879 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385861 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nclln\" (UniqueName: \"kubernetes.io/projected/ec96604b-256b-404b-95d1-5fe5cb00a147-kube-api-access-nclln\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386094 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385888 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-hosts-file\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.386094 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385914 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-tmp-dir\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.386094 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385941 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-system-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386094 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385970 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-kubelet\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386094 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.385996 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:49.386094 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386039 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386094 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386065 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-socket-dir-parent\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386390 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386153 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-k8s-cni-cncf-io\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386390 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386212 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-daemon-config\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386390 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386253 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-multus-certs\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386390 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386291 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbclk\" (UniqueName: \"kubernetes.io/projected/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-kube-api-access-tbclk\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.386390 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386334 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxdcf\" (UniqueName: \"kubernetes.io/projected/03b9a551-4faa-46fb-8d85-9a39e39d9191-kube-api-access-qxdcf\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:49.386390 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386355 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-cnibin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386649 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386419 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-cni-binary-copy\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386649 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386508 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-bin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.386649 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.386537 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-hostroot\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.388066 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.388051 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:49.388136 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.388117 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:06:49.390384 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.390365 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:06:49.392880 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.392858 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.395387 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.395368 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.428438 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.428419 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:06:49.428504 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.428428 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:06:49.428504 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.428426 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:06:49.428504 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.428430 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:06:49.428504 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.428500 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bkplg\"" Nov 03 22:06:49.431062 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.431030 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:06:49.431157 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.431074 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ckrtj\"" Nov 03 22:06:49.431157 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.431074 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:06:49.432932 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.432918 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5mdhj\"" Nov 03 22:06:49.433256 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.433236 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:06:49.433256 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.433249 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:06:49.433337 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.433299 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:06:49.437960 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.437941 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:06:49.438043 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.438029 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:06:49.438080 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.438067 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:06:49.473011 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.472955 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:06:49.487676 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487654 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-kubelet-config\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:49.487754 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487684 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:49.487754 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487703 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.487754 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487718 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-multus-certs\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.487754 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.487885 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487794 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysconfig\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.487885 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.487832 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:49.487957 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487880 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-run\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.487957 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.487890 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs podName:03b9a551-4faa-46fb-8d85-9a39e39d9191 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:49.987872738 +0000 UTC m=+3.067321774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs") pod "network-metrics-daemon-xkldp" (UID: "03b9a551-4faa-46fb-8d85-9a39e39d9191") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:49.487957 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487889 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-multus-certs\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.487957 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487919 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-tuned\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.487957 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487950 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-systemd-units\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.488110 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.487990 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-cni-binary-copy\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488110 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488015 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-bin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488110 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488041 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-sys-fs\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.488110 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488044 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488229 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488120 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c201439-de30-4e86-88ca-c0826c25fe86-host\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.488229 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488149 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-bin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488229 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488179 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vp9t\" (UniqueName: \"kubernetes.io/projected/5c1391f4-b5e9-4e91-88b3-208c56c2a153-kube-api-access-9vp9t\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.488229 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488204 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-host-slash\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.488351 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488239 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23923c80-00d0-4e1b-adbc-89cc9a992709-konnectivity-ca\") pod \"konnectivity-agent-x78ps\" (UID: \"23923c80-00d0-4e1b-adbc-89cc9a992709\") " pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:06:49.488351 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488273 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-netns\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488351 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488295 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-hosts-file\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.488351 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488312 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-var-lib-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.488475 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488358 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-netns\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488475 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488359 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-env-overrides\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.488475 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488394 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-hosts-file\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.488475 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488438 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9dv\" (UniqueName: \"kubernetes.io/projected/6c201439-de30-4e86-88ca-c0826c25fe86-kube-api-access-jk9dv\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.488475 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488472 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-lib-modules\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.488642 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488493 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-k8s-cni-cncf-io\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488642 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488536 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbclk\" (UniqueName: \"kubernetes.io/projected/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-kube-api-access-tbclk\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.488642 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488541 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-k8s-cni-cncf-io\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488642 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488564 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-etc-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.488642 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488570 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-cni-binary-copy\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488642 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488593 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxdcf\" (UniqueName: \"kubernetes.io/projected/03b9a551-4faa-46fb-8d85-9a39e39d9191-kube-api-access-qxdcf\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:49.488816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488640 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-etc-kubernetes\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488725 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23923c80-00d0-4e1b-adbc-89cc9a992709-agent-certs\") pod \"konnectivity-agent-x78ps\" (UID: \"23923c80-00d0-4e1b-adbc-89cc9a992709\") " pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:06:49.488816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488757 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-etc-kubernetes\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.488816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488789 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8hb\" (UniqueName: \"kubernetes.io/projected/87e0c26d-7122-4e83-b875-35a4302cd6e1-kube-api-access-9x8hb\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.488945 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488822 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-slash\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.488945 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488862 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-ovn\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.488945 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488892 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-socket-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.488945 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488925 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.489062 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488951 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-var-lib-kubelet\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.489062 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.488974 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-tmp\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.489062 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-kubelet\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.489062 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489036 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.489182 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489067 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-netd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.489182 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489087 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-config\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.489182 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489119 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.489182 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489136 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovn-node-metrics-cert\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.489182 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489165 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-daemon-config\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489369 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489251 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-system-cni-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.489369 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489278 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-kubernetes\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.489369 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489294 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-conf\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.489369 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489312 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-sys\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.489369 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489328 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-log-socket\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.489369 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489362 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlgzg\" (UniqueName: \"kubernetes.io/projected/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-kube-api-access-qlgzg\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.489554 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489402 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:49.489554 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-cnibin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489554 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-hostroot\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489554 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489539 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-host\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.489690 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489567 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-cnibin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489690 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489589 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-daemon-config\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489690 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489614 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-systemd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.489690 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489643 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-hostroot\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489690 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489673 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-dbus\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:49.489833 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489711 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-multus\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489833 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489747 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-system-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489833 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489772 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-registration-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.489833 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489808 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-multus\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489949 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489837 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c201439-de30-4e86-88ca-c0826c25fe86-serviceca\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.489949 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489865 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-system-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489949 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489863 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.489949 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489912 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-socket-dir-parent\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.489949 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489935 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.490090 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489952 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cnibin\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.490090 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.489988 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.490090 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490007 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-socket-dir-parent\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.490090 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490013 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-modprobe-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.490090 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490041 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.490090 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490070 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-node-log\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.490268 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490093 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-iptables-alerter-script\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.490268 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490110 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nclln\" (UniqueName: \"kubernetes.io/projected/ec96604b-256b-404b-95d1-5fe5cb00a147-kube-api-access-nclln\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.490268 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490144 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-tmp-dir\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.490268 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490237 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.490268 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490260 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-conf-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.490466 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490289 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-device-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.490466 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490335 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.490466 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490340 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-conf-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.490466 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490373 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxdqr\" (UniqueName: \"kubernetes.io/projected/a344707b-bc31-4587-ad1b-5da15d0a4dc9-kube-api-access-wxdqr\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.490466 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490391 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-tmp-dir\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.490466 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490430 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-netns\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.490713 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490475 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-bin\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.490713 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490542 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-kubelet\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.490713 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490573 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-systemd\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.490713 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490619 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-kubelet\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.490713 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490643 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-script-lib\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.490713 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490669 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggwr4\" (UniqueName: \"kubernetes.io/projected/e40ade3f-f022-48c6-b9ed-a219100b4ff2-kube-api-access-ggwr4\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.490956 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.490710 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:49.530369 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.530347 2585 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:06:49.530456 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.530397 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9dqz7" Nov 03 22:06:49.536939 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.536924 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:06:49.537005 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.536990 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:06:49.537765 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537668 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:06:49.537765 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537697 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:06:49.537765 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537722 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:06:49.537765 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537708 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:06:49.537765 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537724 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:06:49.537765 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537761 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:06:49.537765 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537765 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:06:49.538108 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537787 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:06:49.538108 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537809 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dngqk\"" Nov 03 22:06:49.538108 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537854 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:06:49.538108 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.537961 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:06:49.538108 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.538008 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:06:49.538108 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.538017 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:06:49.538108 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.538026 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j54w4\"" Nov 03 22:06:49.538108 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.538020 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8pftz\"" Nov 03 22:06:49.538108 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.538042 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z5g9w\"" Nov 03 22:06:49.538108 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.538106 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:06:49.538399 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.538139 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgskz\"" Nov 03 22:06:49.538399 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.538172 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m27qh\"" Nov 03 22:06:49.547750 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.547730 2585 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:06:49.551429 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.551407 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nclln\" (UniqueName: \"kubernetes.io/projected/ec96604b-256b-404b-95d1-5fe5cb00a147-kube-api-access-nclln\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:06:49.551429 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.551419 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxdcf\" (UniqueName: \"kubernetes.io/projected/03b9a551-4faa-46fb-8d85-9a39e39d9191-kube-api-access-qxdcf\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:49.551557 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.551413 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbclk\" (UniqueName: \"kubernetes.io/projected/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-kube-api-access-tbclk\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.569119 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.569081 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" event={"ID":"56a5599a8af5a3ab98eedd947c314121","Type":"ContainerStarted","Data":"0d0c414cba1b5256629dc7cd6b00f18c908e6f4e6d3ffedd9791bcf723e1df8d"} Nov 03 22:06:49.570110 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.570088 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" event={"ID":"bad3247384fc65111dcedea645af8378","Type":"ContainerStarted","Data":"6bc001e10574ea0052e18de1dcac1a0610084d34cee3c7266a718828f1b0fcda"} Nov 03 22:06:49.590925 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.590902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-kubelet-config\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:49.590990 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.590977 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-kubelet-config\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:49.591048 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591031 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.591101 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591061 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysconfig\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.591101 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-run\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.591202 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591104 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-tuned\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.591202 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591120 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-systemd-units\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.591202 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591140 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-sys-fs\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.591202 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591197 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-sys-fs\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.591382 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591209 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysconfig\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.591382 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591208 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-systemd-units\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.591382 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591256 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-run\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.591382 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591275 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c201439-de30-4e86-88ca-c0826c25fe86-host\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.591382 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591305 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c201439-de30-4e86-88ca-c0826c25fe86-host\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.591382 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591305 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vp9t\" (UniqueName: \"kubernetes.io/projected/5c1391f4-b5e9-4e91-88b3-208c56c2a153-kube-api-access-9vp9t\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.591382 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591335 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-host-slash\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.591382 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591377 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23923c80-00d0-4e1b-adbc-89cc9a992709-konnectivity-ca\") pod \"konnectivity-agent-x78ps\" (UID: \"23923c80-00d0-4e1b-adbc-89cc9a992709\") " pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591388 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-host-slash\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591397 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-var-lib-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591420 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-var-lib-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591432 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-env-overrides\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9dv\" (UniqueName: \"kubernetes.io/projected/6c201439-de30-4e86-88ca-c0826c25fe86-kube-api-access-jk9dv\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591489 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-lib-modules\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591537 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-etc-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591567 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23923c80-00d0-4e1b-adbc-89cc9a992709-agent-certs\") pod \"konnectivity-agent-x78ps\" (UID: \"23923c80-00d0-4e1b-adbc-89cc9a992709\") " pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591589 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591595 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8hb\" (UniqueName: \"kubernetes.io/projected/87e0c26d-7122-4e83-b875-35a4302cd6e1-kube-api-access-9x8hb\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591639 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-slash\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591662 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-etc-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591665 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-ovn\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591699 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-socket-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591709 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-ovn\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591724 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.591816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591745 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-var-lib-kubelet\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591760 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-tmp\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591775 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-kubelet\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591796 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591822 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-netd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-config\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591850 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-socket-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591871 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591885 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-var-lib-kubelet\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovn-node-metrics-cert\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591962 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-lib-modules\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591975 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-slash\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.591980 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-env-overrides\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592016 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-netd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592022 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23923c80-00d0-4e1b-adbc-89cc9a992709-konnectivity-ca\") pod \"konnectivity-agent-x78ps\" (UID: \"23923c80-00d0-4e1b-adbc-89cc9a992709\") " pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592031 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-system-cni-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592061 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-kubernetes\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.592696 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-conf\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592061 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-kubelet\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592094 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592117 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-sys\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592088 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-system-cni-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592139 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-kubernetes\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-log-socket\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592209 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlgzg\" (UniqueName: \"kubernetes.io/projected/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-kube-api-access-qlgzg\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592249 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592259 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592377 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-conf\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592383 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-sys\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592436 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-log-socket\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.592457 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592482 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-host\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.592553 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret podName:5b28053b-cac3-4543-a09b-25915e6b3f0c nodeName:}" failed. No retries permitted until 2026-11-03 22:06:50.092513308 +0000 UTC m=+3.171962342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret") pod "global-pull-secret-syncer-lwk4s" (UID: "5b28053b-cac3-4543-a09b-25915e6b3f0c") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592578 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-systemd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.593564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592597 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-host\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592630 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-systemd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592652 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-dbus\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592701 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-dbus\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592745 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-registration-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c201439-de30-4e86-88ca-c0826c25fe86-serviceca\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592815 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592874 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cnibin\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592892 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-registration-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-modprobe-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592965 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.592969 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593006 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-node-log\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593025 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cnibin\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593037 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-iptables-alerter-script\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.594358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593044 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-device-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593129 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdqr\" (UniqueName: \"kubernetes.io/projected/a344707b-bc31-4587-ad1b-5da15d0a4dc9-kube-api-access-wxdqr\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593159 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-modprobe-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593186 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-netns\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593202 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593216 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-bin\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593244 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-systemd\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593266 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593273 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-script-lib\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggwr4\" (UniqueName: \"kubernetes.io/projected/e40ade3f-f022-48c6-b9ed-a219100b4ff2-kube-api-access-ggwr4\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593337 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593382 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593429 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-netns\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593639 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-node-log\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.594928 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593686 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593814 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593864 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-device-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593900 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-bin\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.593959 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-systemd\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.594005 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-config\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.594052 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-script-lib\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.594172 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c201439-de30-4e86-88ca-c0826c25fe86-serviceca\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.594276 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-iptables-alerter-script\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.594431 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-tmp\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.594474 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-tuned\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.595428 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.594682 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23923c80-00d0-4e1b-adbc-89cc9a992709-agent-certs\") pod \"konnectivity-agent-x78ps\" (UID: \"23923c80-00d0-4e1b-adbc-89cc9a992709\") " pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:06:49.595856 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.595839 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovn-node-metrics-cert\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.652777 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.652750 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:49.652891 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.652783 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:49.652891 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.652797 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w4gz5 for pod openshift-network-diagnostics/network-check-target-9bqcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:49.652891 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.652882 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5 podName:b595e184-27c8-4ffc-b50c-9764f013ef04 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:50.15286148 +0000 UTC m=+3.232310512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w4gz5" (UniqueName: "kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5") pod "network-check-target-9bqcf" (UID: "b595e184-27c8-4ffc-b50c-9764f013ef04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:49.653959 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.653930 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9dv\" (UniqueName: \"kubernetes.io/projected/6c201439-de30-4e86-88ca-c0826c25fe86-kube-api-access-jk9dv\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.654072 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.654039 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlgzg\" (UniqueName: \"kubernetes.io/projected/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-kube-api-access-qlgzg\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.654205 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.654180 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8hb\" (UniqueName: \"kubernetes.io/projected/87e0c26d-7122-4e83-b875-35a4302cd6e1-kube-api-access-9x8hb\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.654262 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.654237 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vp9t\" (UniqueName: \"kubernetes.io/projected/5c1391f4-b5e9-4e91-88b3-208c56c2a153-kube-api-access-9vp9t\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.654927 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.654904 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdqr\" (UniqueName: \"kubernetes.io/projected/a344707b-bc31-4587-ad1b-5da15d0a4dc9-kube-api-access-wxdqr\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.655405 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.655385 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggwr4\" (UniqueName: \"kubernetes.io/projected/e40ade3f-f022-48c6-b9ed-a219100b4ff2-kube-api-access-ggwr4\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.685025 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.685008 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-796dw" Nov 03 22:06:49.691414 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.691397 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:06:49.695105 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.695084 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec96604b_256b_404b_95d1_5fe5cb00a147.slice/crio-613bda4db336d7790de31f19360018b21b4b764209444ee062d2f72aed278abc WatchSource:0}: Error finding container 613bda4db336d7790de31f19360018b21b4b764209444ee062d2f72aed278abc: Status 404 returned error can't find the container with id 613bda4db336d7790de31f19360018b21b4b764209444ee062d2f72aed278abc Nov 03 22:06:49.698525 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.698501 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:06:49.700819 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.700789 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fdc02e_6d8f_4e82_a8a4_9f295b64285d.slice/crio-517213ad3a35d48e0b4d1cbb0f2ba917e4d4d408dfc37bce2a3b9f9409a7d143 WatchSource:0}: Error finding container 517213ad3a35d48e0b4d1cbb0f2ba917e4d4d408dfc37bce2a3b9f9409a7d143: Status 404 returned error can't find the container with id 517213ad3a35d48e0b4d1cbb0f2ba917e4d4d408dfc37bce2a3b9f9409a7d143 Nov 03 22:06:49.703971 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.703953 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:06:49.708045 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.708026 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd82c8e7_5bc3_4cb7_be68_a519c025f61f.slice/crio-824eadf4320ba290af7854d552fe68b2a4ac02727ca5e7fd8d16037d3e54c53f WatchSource:0}: Error finding container 824eadf4320ba290af7854d552fe68b2a4ac02727ca5e7fd8d16037d3e54c53f: Status 404 returned error can't find the container with id 824eadf4320ba290af7854d552fe68b2a4ac02727ca5e7fd8d16037d3e54c53f Nov 03 22:06:49.713319 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.713298 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c201439_de30_4e86_88ca_c0826c25fe86.slice/crio-89c757a5702c30100e8ae8993aa9377cf1ddcbfda0678b6905d6e97e94a07355 WatchSource:0}: Error finding container 89c757a5702c30100e8ae8993aa9377cf1ddcbfda0678b6905d6e97e94a07355: Status 404 returned error can't find the container with id 89c757a5702c30100e8ae8993aa9377cf1ddcbfda0678b6905d6e97e94a07355 Nov 03 22:06:49.720670 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.720650 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:06:49.724397 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.724339 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:06:49.726951 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.726931 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:06:49.730675 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.730650 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda344707b_bc31_4587_ad1b_5da15d0a4dc9.slice/crio-f7c323ccb850a139e16466b1b38bc4cf33f545d0865cc08ddb6aaa08581bb9b3 WatchSource:0}: Error finding container f7c323ccb850a139e16466b1b38bc4cf33f545d0865cc08ddb6aaa08581bb9b3: Status 404 returned error can't find the container with id f7c323ccb850a139e16466b1b38bc4cf33f545d0865cc08ddb6aaa08581bb9b3 Nov 03 22:06:49.735600 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.734839 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1391f4_b5e9_4e91_88b3_208c56c2a153.slice/crio-648fef1e7f821744d10df75f0b58cbcf3c3186a55452ba7ed725afd35a611762 WatchSource:0}: Error finding container 648fef1e7f821744d10df75f0b58cbcf3c3186a55452ba7ed725afd35a611762: Status 404 returned error can't find the container with id 648fef1e7f821744d10df75f0b58cbcf3c3186a55452ba7ed725afd35a611762 Nov 03 22:06:49.737101 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.737081 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23923c80_00d0_4e1b_adbc_89cc9a992709.slice/crio-b2e3d6ffc8dedbdf6d54389ed6c94a671c7f96b9f3194c240958151422e18d54 WatchSource:0}: Error finding container b2e3d6ffc8dedbdf6d54389ed6c94a671c7f96b9f3194c240958151422e18d54: Status 404 returned error can't find the container with id b2e3d6ffc8dedbdf6d54389ed6c94a671c7f96b9f3194c240958151422e18d54 Nov 03 22:06:49.747952 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.747932 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:06:49.752294 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.752276 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:06:49.759405 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.759384 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e0c26d_7122_4e83_b875_35a4302cd6e1.slice/crio-a1ffd7459a3c94e519a3c1e82dc965b8c757c2ecd19c1956aad2a0bcf579e376 WatchSource:0}: Error finding container a1ffd7459a3c94e519a3c1e82dc965b8c757c2ecd19c1956aad2a0bcf579e376: Status 404 returned error can't find the container with id a1ffd7459a3c94e519a3c1e82dc965b8c757c2ecd19c1956aad2a0bcf579e376 Nov 03 22:06:49.761282 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:06:49.761260 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40ade3f_f022_48c6_b9ed_a219100b4ff2.slice/crio-8c6561d8dab64ac4f2a448bbb40c9cecd1c3cf2696d7af9e311496d39570f435 WatchSource:0}: Error finding container 8c6561d8dab64ac4f2a448bbb40c9cecd1c3cf2696d7af9e311496d39570f435: Status 404 returned error can't find the container with id 8c6561d8dab64ac4f2a448bbb40c9cecd1c3cf2696d7af9e311496d39570f435 Nov 03 22:06:49.996489 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:49.996409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:49.996650 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.996552 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:49.996650 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:49.996603 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs podName:03b9a551-4faa-46fb-8d85-9a39e39d9191 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:50.996589307 +0000 UTC m=+4.076038327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs") pod "network-metrics-daemon-xkldp" (UID: "03b9a551-4faa-46fb-8d85-9a39e39d9191") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:50.097871 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.097834 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:50.098039 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:50.097954 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:50.098039 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:50.098008 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret podName:5b28053b-cac3-4543-a09b-25915e6b3f0c nodeName:}" failed. No retries permitted until 2026-11-03 22:06:51.097994541 +0000 UTC m=+4.177443557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret") pod "global-pull-secret-syncer-lwk4s" (UID: "5b28053b-cac3-4543-a09b-25915e6b3f0c") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:50.199453 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.199418 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:50.199623 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:50.199600 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:50.199693 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:50.199628 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:50.199693 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:50.199639 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w4gz5 for pod openshift-network-diagnostics/network-check-target-9bqcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:50.199770 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:50.199705 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5 podName:b595e184-27c8-4ffc-b50c-9764f013ef04 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:51.199683769 +0000 UTC m=+4.279132790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4gz5" (UniqueName: "kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5") pod "network-check-target-9bqcf" (UID: "b595e184-27c8-4ffc-b50c-9764f013ef04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:50.533969 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.533831 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:01:50 +0000 UTC" deadline="2028-07-19 07:17:49.790676054 +0000 UTC" Nov 03 22:06:50.533969 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.533901 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14961h10m59.2567796s" Nov 03 22:06:50.566314 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.566271 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:50.566314 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.566315 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:50.566550 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:50.566422 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:06:50.566644 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:50.566619 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:06:50.573556 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.573526 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-796dw" event={"ID":"ec96604b-256b-404b-95d1-5fe5cb00a147","Type":"ContainerStarted","Data":"613bda4db336d7790de31f19360018b21b4b764209444ee062d2f72aed278abc"} Nov 03 22:06:50.575114 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.575031 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" event={"ID":"87e0c26d-7122-4e83-b875-35a4302cd6e1","Type":"ContainerStarted","Data":"a1ffd7459a3c94e519a3c1e82dc965b8c757c2ecd19c1956aad2a0bcf579e376"} Nov 03 22:06:50.576618 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.576596 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" event={"ID":"5c1391f4-b5e9-4e91-88b3-208c56c2a153","Type":"ContainerStarted","Data":"648fef1e7f821744d10df75f0b58cbcf3c3186a55452ba7ed725afd35a611762"} Nov 03 22:06:50.578070 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.578040 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" event={"ID":"a344707b-bc31-4587-ad1b-5da15d0a4dc9","Type":"ContainerStarted","Data":"f7c323ccb850a139e16466b1b38bc4cf33f545d0865cc08ddb6aaa08581bb9b3"} Nov 03 22:06:50.579396 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.579370 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d8jgp" event={"ID":"13fdc02e-6d8f-4e82-a8a4-9f295b64285d","Type":"ContainerStarted","Data":"517213ad3a35d48e0b4d1cbb0f2ba917e4d4d408dfc37bce2a3b9f9409a7d143"} Nov 03 22:06:50.580992 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.580888 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" event={"ID":"e40ade3f-f022-48c6-b9ed-a219100b4ff2","Type":"ContainerStarted","Data":"8c6561d8dab64ac4f2a448bbb40c9cecd1c3cf2696d7af9e311496d39570f435"} Nov 03 22:06:50.582456 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.582430 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x78ps" event={"ID":"23923c80-00d0-4e1b-adbc-89cc9a992709","Type":"ContainerStarted","Data":"b2e3d6ffc8dedbdf6d54389ed6c94a671c7f96b9f3194c240958151422e18d54"} Nov 03 22:06:50.584552 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.584527 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v6n4q" event={"ID":"6c201439-de30-4e86-88ca-c0826c25fe86","Type":"ContainerStarted","Data":"89c757a5702c30100e8ae8993aa9377cf1ddcbfda0678b6905d6e97e94a07355"} Nov 03 22:06:50.586012 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:50.585986 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2swns" event={"ID":"fd82c8e7-5bc3-4cb7-be68-a519c025f61f","Type":"ContainerStarted","Data":"824eadf4320ba290af7854d552fe68b2a4ac02727ca5e7fd8d16037d3e54c53f"} Nov 03 22:06:51.006794 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:51.006710 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:51.006958 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:51.006908 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:51.007022 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:51.006977 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs podName:03b9a551-4faa-46fb-8d85-9a39e39d9191 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:53.006956939 +0000 UTC m=+6.086405960 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs") pod "network-metrics-daemon-xkldp" (UID: "03b9a551-4faa-46fb-8d85-9a39e39d9191") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:51.107801 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:51.107769 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:51.107992 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:51.107918 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:51.108072 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:51.108017 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret podName:5b28053b-cac3-4543-a09b-25915e6b3f0c nodeName:}" failed. No retries permitted until 2026-11-03 22:06:53.107984023 +0000 UTC m=+6.187433044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret") pod "global-pull-secret-syncer-lwk4s" (UID: "5b28053b-cac3-4543-a09b-25915e6b3f0c") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:51.208298 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:51.208260 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:51.208491 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:51.208449 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:51.208491 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:51.208473 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:51.208491 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:51.208486 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w4gz5 for pod openshift-network-diagnostics/network-check-target-9bqcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:51.208686 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:51.208545 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5 podName:b595e184-27c8-4ffc-b50c-9764f013ef04 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:53.208527024 +0000 UTC m=+6.287976056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4gz5" (UniqueName: "kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5") pod "network-check-target-9bqcf" (UID: "b595e184-27c8-4ffc-b50c-9764f013ef04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:51.534758 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:51.534704 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:01:50 +0000 UTC" deadline="2028-04-11 08:00:16.242716344 +0000 UTC" Nov 03 22:06:51.534758 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:51.534745 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12585h53m24.707976685s" Nov 03 22:06:51.571289 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:51.571226 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:51.571497 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:51.571358 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:06:52.566172 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:52.566108 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:52.566661 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:52.566286 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:06:52.566661 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:52.566386 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:52.566661 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:52.566481 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:06:53.027984 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:53.027920 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:53.028146 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:53.028118 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:53.028211 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:53.028183 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs podName:03b9a551-4faa-46fb-8d85-9a39e39d9191 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:57.028163896 +0000 UTC m=+10.107612926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs") pod "network-metrics-daemon-xkldp" (UID: "03b9a551-4faa-46fb-8d85-9a39e39d9191") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:53.129070 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:53.128760 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:53.129070 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:53.128961 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:53.129070 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:53.129031 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret podName:5b28053b-cac3-4543-a09b-25915e6b3f0c nodeName:}" failed. No retries permitted until 2026-11-03 22:06:57.129012602 +0000 UTC m=+10.208461640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret") pod "global-pull-secret-syncer-lwk4s" (UID: "5b28053b-cac3-4543-a09b-25915e6b3f0c") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:53.229633 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:53.229511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:53.229795 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:53.229732 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:53.229795 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:53.229756 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:53.229795 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:53.229769 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w4gz5 for pod openshift-network-diagnostics/network-check-target-9bqcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:53.229963 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:53.229833 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5 podName:b595e184-27c8-4ffc-b50c-9764f013ef04 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:57.22981459 +0000 UTC m=+10.309263613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4gz5" (UniqueName: "kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5") pod "network-check-target-9bqcf" (UID: "b595e184-27c8-4ffc-b50c-9764f013ef04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:53.567823 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:53.567784 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:53.568480 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:53.567937 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:06:54.566158 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:54.566125 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:54.566330 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:54.566271 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:06:54.566401 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:54.566365 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:54.566478 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:54.566458 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:06:54.639130 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:54.639069 2585 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="bc37e8d5fc5b7d8c8f77121b16b8e58442ddb0b9e676b0901ef445471a7c7bc6" exitCode=0 Nov 03 22:06:54.639688 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:54.639222 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" event={"ID":"5c1391f4-b5e9-4e91-88b3-208c56c2a153","Type":"ContainerDied","Data":"bc37e8d5fc5b7d8c8f77121b16b8e58442ddb0b9e676b0901ef445471a7c7bc6"} Nov 03 22:06:54.648944 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:54.648915 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" event={"ID":"56a5599a8af5a3ab98eedd947c314121","Type":"ContainerStarted","Data":"03770f0f32152907d9f07a0cc90677dba127cfa0f5bd1b16804bff1ef85bd5a0"} Nov 03 22:06:54.655667 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:54.655104 2585 generic.go:356] "Generic (PLEG): container finished" podID="bad3247384fc65111dcedea645af8378" containerID="e666d5967bd143c4b2ac1f278b1c6d082583efd983912bf8e411f92fb70b1e68" exitCode=0 Nov 03 22:06:54.655667 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:54.655144 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" event={"ID":"bad3247384fc65111dcedea645af8378","Type":"ContainerDied","Data":"e666d5967bd143c4b2ac1f278b1c6d082583efd983912bf8e411f92fb70b1e68"} Nov 03 22:06:55.569819 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:55.569499 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:55.569819 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:55.569664 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:06:55.662668 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:55.661765 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x78ps" event={"ID":"23923c80-00d0-4e1b-adbc-89cc9a992709","Type":"ContainerStarted","Data":"4f2eef33631806074213fafd4328e210f5261e9189e21736034dba9d6f2f6844"} Nov 03 22:06:55.668837 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:55.668743 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" event={"ID":"bad3247384fc65111dcedea645af8378","Type":"ContainerStarted","Data":"cec161917833f09142d96c90b223b5883bf8537e0736b7a394789028f9121521"} Nov 03 22:06:55.784810 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:55.784742 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" podStartSLOduration=6.784712513 podStartE2EDuration="6.784712513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:55.119667373 +0000 UTC m=+8.199116418" watchObservedRunningTime="2026-11-03 22:06:55.784712513 +0000 UTC m=+8.864161563" Nov 03 22:06:55.906566 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:55.906456 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x78ps" podStartSLOduration=2.495264962 podStartE2EDuration="7.906433436s" totalImagesPullingTime="5.411168474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:55.785025124 +0000 UTC m=+8.864474164" watchObservedRunningTime="2026-11-03 22:06:55.906433436 +0000 UTC m=+8.985882477" Nov 03 22:06:55.906759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:55.906701 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" podStartSLOduration=6.906691638 podStartE2EDuration="6.906691638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:55.905832941 +0000 UTC m=+8.985281982" watchObservedRunningTime="2026-11-03 22:06:55.906691638 +0000 UTC m=+8.986140680" Nov 03 22:06:56.566343 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:56.566250 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:56.566539 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:56.566371 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:06:56.566539 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:56.566437 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:56.566539 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:56.566505 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:06:56.671912 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:56.671872 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v6n4q" event={"ID":"6c201439-de30-4e86-88ca-c0826c25fe86","Type":"ContainerStarted","Data":"f58043897ff50f51285d406642cecd7745e20e1990083d2d3e4c737d52c329b4"} Nov 03 22:06:56.795957 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:56.795890 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v6n4q" podStartSLOduration=2.517949808 podStartE2EDuration="8.795870526s" totalImagesPullingTime="6.277920718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:56.795518565 +0000 UTC m=+9.874967607" watchObservedRunningTime="2026-11-03 22:06:56.795870526 +0000 UTC m=+9.875319569" Nov 03 22:06:57.062315 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:57.062235 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:57.062480 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:57.062410 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:57.062561 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:57.062476 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs podName:03b9a551-4faa-46fb-8d85-9a39e39d9191 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.06245618 +0000 UTC m=+18.141905199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs") pod "network-metrics-daemon-xkldp" (UID: "03b9a551-4faa-46fb-8d85-9a39e39d9191") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:57.163557 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:57.163506 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:57.163708 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:57.163676 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:57.163776 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:57.163765 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret podName:5b28053b-cac3-4543-a09b-25915e6b3f0c nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.163742173 +0000 UTC m=+18.243191215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret") pod "global-pull-secret-syncer-lwk4s" (UID: "5b28053b-cac3-4543-a09b-25915e6b3f0c") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:57.264401 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:57.264360 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:57.264597 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:57.264582 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:57.264659 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:57.264606 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:57.264659 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:57.264620 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w4gz5 for pod openshift-network-diagnostics/network-check-target-9bqcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:57.264739 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:57.264678 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5 podName:b595e184-27c8-4ffc-b50c-9764f013ef04 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.264659076 +0000 UTC m=+18.344108107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4gz5" (UniqueName: "kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5") pod "network-check-target-9bqcf" (UID: "b595e184-27c8-4ffc-b50c-9764f013ef04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:57.570191 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:57.570148 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:57.570387 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:57.570284 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:06:57.677724 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:57.677604 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" event={"ID":"87e0c26d-7122-4e83-b875-35a4302cd6e1","Type":"ContainerStarted","Data":"79ffb32499281f8241f34bf78f617847349609721426d857636c32ba01755807"} Nov 03 22:06:58.016473 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:58.016394 2585 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:06:58.022243 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:58.022167 2585 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:06:58.567424 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:58.567140 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:06:58.567424 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:58.567196 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:06:58.567424 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:58.567301 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:06:58.567424 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:58.567369 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:06:58.683215 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:58.683168 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-796dw" event={"ID":"ec96604b-256b-404b-95d1-5fe5cb00a147","Type":"ContainerStarted","Data":"bc1e2a2ebf8b4ba064ae6e0857b9143c0b8d1be53f37152bf5ac4189a27223f9"} Nov 03 22:06:59.495368 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:59.495339 2585 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:06:59.566241 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:59.566198 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:06:59.566402 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:06:59.566367 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:06:59.689742 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:59.689655 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" event={"ID":"87e0c26d-7122-4e83-b875-35a4302cd6e1","Type":"ContainerStarted","Data":"31d1c3eb3c761f89f4c81cb1394d5758ce57b8e46592beb39ca3bc2e82a35ee4"} Nov 03 22:06:59.691694 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:59.691640 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d8jgp" event={"ID":"13fdc02e-6d8f-4e82-a8a4-9f295b64285d","Type":"ContainerStarted","Data":"5f78296de09475f9575900a53e6e680567bde3ebbf2dbed01fc74f5b2dfaa1b0"} Nov 03 22:06:59.819560 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:59.819500 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-d8jgp" podStartSLOduration=2.535281284 podStartE2EDuration="11.819481161s" totalImagesPullingTime="9.284199877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:59.819322444 +0000 UTC m=+12.898771507" watchObservedRunningTime="2026-11-03 22:06:59.819481161 +0000 UTC m=+12.898930203" Nov 03 22:06:59.820610 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:06:59.820368 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-796dw" podStartSLOduration=3.014872521 podStartE2EDuration="11.820356866s" totalImagesPullingTime="8.805484345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:58.808031382 +0000 UTC m=+11.887480424" watchObservedRunningTime="2026-11-03 22:06:59.820356866 +0000 UTC m=+12.899805907" Nov 03 22:07:00.468715 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:00.468554 2585 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:06:59.495363544Z","UUID":"7ea3bcfc-1cd0-4405-a9bc-04e8f813e5ef","Handler":null,"Name":"","Endpoint":""} Nov 03 22:07:00.470191 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:00.470169 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:07:00.470191 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:00.470198 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:07:00.566307 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:00.566274 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:00.566464 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:00.566326 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:00.566464 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:00.566400 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:00.566662 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:00.566539 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:00.697991 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:00.697957 2585 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="8423b86d5f760e1a1f5f5ac810b31284257250c41bf8aacbec30792ccc943749" exitCode=0 Nov 03 22:07:00.698462 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:00.698055 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" event={"ID":"5c1391f4-b5e9-4e91-88b3-208c56c2a153","Type":"ContainerDied","Data":"8423b86d5f760e1a1f5f5ac810b31284257250c41bf8aacbec30792ccc943749"} Nov 03 22:07:01.565808 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:01.565783 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:01.565917 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:01.565896 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:01.702223 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:01.702185 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2swns" event={"ID":"fd82c8e7-5bc3-4cb7-be68-a519c025f61f","Type":"ContainerStarted","Data":"8ddf4c6ce894fc238921608b525f522de45b83fdedf76c7af18f4d2e6ed9fbef"} Nov 03 22:07:01.705256 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:01.705228 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" event={"ID":"87e0c26d-7122-4e83-b875-35a4302cd6e1","Type":"ContainerStarted","Data":"a9d95925920040afb11c4d40fa91eae32f25f223f0ee1ad857940ca3a4d44748"} Nov 03 22:07:01.706781 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:01.706750 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" event={"ID":"a344707b-bc31-4587-ad1b-5da15d0a4dc9","Type":"ContainerStarted","Data":"cdcdf2628916f99395f3dfef6c1a4a9c3d26cc1ded87fbb0396ecc59a07177a6"} Nov 03 22:07:01.931097 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:01.931002 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" podStartSLOduration=2.544193083 podStartE2EDuration="13.930982614s" totalImagesPullingTime="11.386789531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:01.930464749 +0000 UTC m=+15.009913785" watchObservedRunningTime="2026-11-03 22:07:01.930982614 +0000 UTC m=+15.010431652" Nov 03 22:07:01.931274 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:01.931246 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2swns" podStartSLOduration=4.461449294 podStartE2EDuration="13.931238263s" totalImagesPullingTime="9.469788969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:01.814606095 +0000 UTC m=+14.894055131" watchObservedRunningTime="2026-11-03 22:07:01.931238263 +0000 UTC m=+15.010687303" Nov 03 22:07:02.045637 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:02.045584 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" podStartSLOduration=2.258756484 podStartE2EDuration="14.045569637s" totalImagesPullingTime="11.786813153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:02.045419824 +0000 UTC m=+15.124868860" watchObservedRunningTime="2026-11-03 22:07:02.045569637 +0000 UTC m=+15.125018666" Nov 03 22:07:02.566048 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:02.566010 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:02.566215 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:02.566052 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:02.566215 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:02.566123 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:02.566322 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:02.566269 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:02.711972 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:02.711897 2585 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="f42981f8874b8843f23b1c8b24bae33f483d3286ba305e368c72ae7955a619a7" exitCode=0 Nov 03 22:07:02.712637 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:02.712000 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" event={"ID":"5c1391f4-b5e9-4e91-88b3-208c56c2a153","Type":"ContainerDied","Data":"f42981f8874b8843f23b1c8b24bae33f483d3286ba305e368c72ae7955a619a7"} Nov 03 22:07:03.566489 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:03.566453 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:03.566678 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:03.566596 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:04.566371 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:04.566346 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:04.566865 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:04.566381 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:04.566865 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:04.566473 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:04.566865 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:04.566588 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:04.719381 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:04.719288 2585 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="d842b3ce486706bc98522cae561e522dfd452f5cedd7c53f4d2fc6cb53a425cb" exitCode=0 Nov 03 22:07:04.719381 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:04.719341 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" event={"ID":"5c1391f4-b5e9-4e91-88b3-208c56c2a153","Type":"ContainerDied","Data":"d842b3ce486706bc98522cae561e522dfd452f5cedd7c53f4d2fc6cb53a425cb"} Nov 03 22:07:05.127224 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:05.126966 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:05.127364 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:05.127105 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:07:05.127364 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:05.127304 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs podName:03b9a551-4faa-46fb-8d85-9a39e39d9191 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:21.127288817 +0000 UTC m=+34.206737833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs") pod "network-metrics-daemon-xkldp" (UID: "03b9a551-4faa-46fb-8d85-9a39e39d9191") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:07:05.227703 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:05.227668 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:05.227852 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:05.227782 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:07:05.227852 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:05.227844 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret podName:5b28053b-cac3-4543-a09b-25915e6b3f0c nodeName:}" failed. No retries permitted until 2026-11-03 22:07:21.227826791 +0000 UTC m=+34.307275823 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret") pod "global-pull-secret-syncer-lwk4s" (UID: "5b28053b-cac3-4543-a09b-25915e6b3f0c") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:07:05.328771 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:05.328743 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:05.328914 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:05.328896 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:07:05.328961 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:05.328919 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:07:05.328961 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:05.328928 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w4gz5 for pod openshift-network-diagnostics/network-check-target-9bqcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:07:05.329023 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:05.328975 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5 podName:b595e184-27c8-4ffc-b50c-9764f013ef04 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:21.328959068 +0000 UTC m=+34.408408097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4gz5" (UniqueName: "kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5") pod "network-check-target-9bqcf" (UID: "b595e184-27c8-4ffc-b50c-9764f013ef04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:07:05.566753 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:05.566681 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:05.567070 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:05.566810 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:06.566324 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:06.566250 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:06.566533 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:06.566405 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:06.566533 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:06.566467 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:06.566662 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:06.566604 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:06.726573 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:06.726541 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" event={"ID":"e40ade3f-f022-48c6-b9ed-a219100b4ff2","Type":"ContainerStarted","Data":"22914119a8fec2d3f94c988b268f6bfb45fdd5b3f229ef6bb60663576352083c"} Nov 03 22:07:06.727065 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:06.726608 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" event={"ID":"e40ade3f-f022-48c6-b9ed-a219100b4ff2","Type":"ContainerStarted","Data":"2d6b16d9c12fb38857f1b161d4de69d8c25edac2feac45b4f2eed4d9a31f6e3a"} Nov 03 22:07:06.727065 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:06.726637 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" event={"ID":"e40ade3f-f022-48c6-b9ed-a219100b4ff2","Type":"ContainerStarted","Data":"f510790aec5dbb3c35f7846ec5aee4634abcf65e047b1e4b14f97ec5cbebcfd3"} Nov 03 22:07:06.727065 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:06.726669 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" event={"ID":"e40ade3f-f022-48c6-b9ed-a219100b4ff2","Type":"ContainerStarted","Data":"b1c61015b95df6ae8be0eea57c1b15b7ddeb67d847e1ffc5304369035810b1e4"} Nov 03 22:07:06.727065 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:06.726707 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" event={"ID":"e40ade3f-f022-48c6-b9ed-a219100b4ff2","Type":"ContainerStarted","Data":"bc9712e080f4f5c1474ba624257ec698bfd343c2b13ebca500ea56e920659e96"} Nov 03 22:07:06.727065 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:06.726755 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" event={"ID":"e40ade3f-f022-48c6-b9ed-a219100b4ff2","Type":"ContainerStarted","Data":"2a2b045b193afec2e939ab949efda1ffc2774ec35547bd571e05cfb836c8cef1"} Nov 03 22:07:07.567309 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:07.567273 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:07.567523 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:07.567350 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:07.730671 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:07.730637 2585 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="5c694f3808b07888598268d29d297242557f374aea8a58fc17e6901c43776a37" exitCode=0 Nov 03 22:07:07.731219 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:07.730691 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" event={"ID":"5c1391f4-b5e9-4e91-88b3-208c56c2a153","Type":"ContainerDied","Data":"5c694f3808b07888598268d29d297242557f374aea8a58fc17e6901c43776a37"} Nov 03 22:07:08.566734 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:08.566657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:08.566870 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:08.566780 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:08.566870 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:08.566825 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:08.566936 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:08.566894 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:08.735762 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:08.735724 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" event={"ID":"e40ade3f-f022-48c6-b9ed-a219100b4ff2","Type":"ContainerStarted","Data":"baedb71fd16a2906b5c578931de0ac82bae43b4cb8a5210708205c9c7b9205c1"} Nov 03 22:07:08.738759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:08.738727 2585 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="cbf42dee22b994ca265750a738a1c984b805dd8d86de27f7247da829fba84252" exitCode=0 Nov 03 22:07:08.738880 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:08.738763 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" event={"ID":"5c1391f4-b5e9-4e91-88b3-208c56c2a153","Type":"ContainerDied","Data":"cbf42dee22b994ca265750a738a1c984b805dd8d86de27f7247da829fba84252"} Nov 03 22:07:09.565975 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:09.565944 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:09.566120 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:09.566060 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:09.744560 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:09.744524 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" event={"ID":"5c1391f4-b5e9-4e91-88b3-208c56c2a153","Type":"ContainerStarted","Data":"3d5d8fa06d8e2feb4580ae85533f2a4cb1845ac92ec01961ee20a89eda2898c4"} Nov 03 22:07:09.864315 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:09.864228 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v7c9n" podStartSLOduration=7.648108135 podStartE2EDuration="21.86421476s" totalImagesPullingTime="14.216106625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:09.864029193 +0000 UTC m=+22.943478241" watchObservedRunningTime="2026-11-03 22:07:09.86421476 +0000 UTC m=+22.943663795" Nov 03 22:07:10.566047 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:10.565785 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:10.566047 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:10.565785 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:10.566272 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:10.566207 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:10.566374 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:10.566346 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:10.750536 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:10.750487 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" event={"ID":"e40ade3f-f022-48c6-b9ed-a219100b4ff2","Type":"ContainerStarted","Data":"4959c5c92bff2d2d004e57f76ca3a9d5661d64c8348fec3b88b318fab4918096"} Nov 03 22:07:10.751032 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:10.750987 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:07:10.751098 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:10.751060 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:07:10.751131 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:10.751109 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:07:10.766189 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:10.766163 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:07:10.766308 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:10.766266 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:07:10.956234 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:10.956136 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" podStartSLOduration=6.976143724 podStartE2EDuration="22.956124126s" totalImagesPullingTime="15.979980402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:10.92292198 +0000 UTC m=+24.002371016" watchObservedRunningTime="2026-11-03 22:07:10.956124126 +0000 UTC m=+24.035573161" Nov 03 22:07:11.566361 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:11.566332 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:11.566612 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:11.566436 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:12.566582 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:12.566553 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:12.566582 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:12.566582 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:12.566973 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:12.566659 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:12.566973 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:12.566776 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:13.081570 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:13.081544 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d8jgp_13fdc02e-6d8f-4e82-a8a4-9f295b64285d/dns-node-resolver/0.log" Nov 03 22:07:13.565888 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:13.565815 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:13.566035 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:13.565935 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:13.847137 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:13.847071 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v6n4q_6c201439-de30-4e86-88ca-c0826c25fe86/node-ca/0.log" Nov 03 22:07:14.566539 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:14.566509 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:14.566708 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:14.566618 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:14.566708 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:14.566668 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:14.566831 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:14.566749 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:15.566191 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:15.566158 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:15.566660 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:15.566284 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:16.566564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:16.566542 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:16.566940 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:16.566551 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:16.566940 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:16.566640 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:16.566940 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:16.566709 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:17.567053 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:17.567018 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:17.567437 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:17.567108 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:18.430219 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:18.430187 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:07:18.430398 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:18.430308 2585 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:07:18.430700 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:18.430685 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:07:18.565847 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:18.565820 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:18.565847 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:18.565850 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:18.566036 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:18.565918 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:18.566036 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:18.565965 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:19.566593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:19.566561 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:19.566970 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:19.566668 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:20.566206 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:20.566172 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:20.566206 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:20.566207 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:20.566403 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:20.566283 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:20.566444 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:20.566413 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:21.146343 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:21.146312 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:21.146784 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:21.146458 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:07:21.146784 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:21.146535 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs podName:03b9a551-4faa-46fb-8d85-9a39e39d9191 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:53.146519844 +0000 UTC m=+66.225968871 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs") pod "network-metrics-daemon-xkldp" (UID: "03b9a551-4faa-46fb-8d85-9a39e39d9191") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:07:21.247232 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:21.247205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:21.247385 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:21.247298 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:07:21.247385 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:21.247344 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret podName:5b28053b-cac3-4543-a09b-25915e6b3f0c nodeName:}" failed. No retries permitted until 2026-11-03 22:07:53.247332079 +0000 UTC m=+66.326781093 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret") pod "global-pull-secret-syncer-lwk4s" (UID: "5b28053b-cac3-4543-a09b-25915e6b3f0c") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:07:21.347909 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:21.347882 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:21.348051 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:21.347998 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:07:21.348051 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:21.348010 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:07:21.348051 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:21.348019 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w4gz5 for pod openshift-network-diagnostics/network-check-target-9bqcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:07:21.348150 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:21.348060 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5 podName:b595e184-27c8-4ffc-b50c-9764f013ef04 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:53.348048688 +0000 UTC m=+66.427497702 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w4gz5" (UniqueName: "kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5") pod "network-check-target-9bqcf" (UID: "b595e184-27c8-4ffc-b50c-9764f013ef04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:07:21.566255 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:21.566170 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:21.566418 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:21.566292 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:22.565902 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:22.565875 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:22.566272 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:22.565875 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:22.566272 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:22.565965 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:22.566272 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:22.566077 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:23.566415 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:23.566384 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:23.566780 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:23.566519 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:24.566636 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:24.566607 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:24.566636 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:24.566642 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:24.567037 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:24.566719 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:24.567037 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:24.566764 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:25.566014 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:25.565987 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:25.566185 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:25.566081 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:26.566193 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:26.566164 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:26.566597 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:26.566164 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:26.566597 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:26.566260 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:26.566597 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:26.566338 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:27.568543 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:27.568517 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:27.568951 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:27.568605 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:28.566570 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:28.566541 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:28.566570 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:28.566574 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:28.566754 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:28.566651 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:28.566796 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:28.566771 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:29.566597 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:29.566569 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:29.566961 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:29.566683 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:29.992594 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:29.992564 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9bqcf"] Nov 03 22:07:29.992714 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:29.992702 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:29.992803 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:29.992783 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:29.994554 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:29.994531 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lwk4s"] Nov 03 22:07:29.994694 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:29.994676 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:29.994812 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:29.994790 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:29.995450 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:29.995432 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkldp"] Nov 03 22:07:29.995548 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:29.995537 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:29.995622 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:29.995607 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:31.091383 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:31.091225 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d8jgp_13fdc02e-6d8f-4e82-a8a4-9f295b64285d/dns-node-resolver/0.log" Nov 03 22:07:31.566711 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:31.566678 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:31.566711 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:31.566710 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:31.566900 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:31.566678 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:31.566900 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:31.566803 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:31.566900 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:31.566885 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:31.566995 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:31.566934 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:32.291969 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:32.291941 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v6n4q_6c201439-de30-4e86-88ca-c0826c25fe86/node-ca/0.log" Nov 03 22:07:33.566279 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:33.566249 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:33.566279 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:33.566266 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:33.566746 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:33.566255 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:33.566746 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:33.566358 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkldp" podUID="03b9a551-4faa-46fb-8d85-9a39e39d9191" Nov 03 22:07:33.566746 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:33.566415 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bqcf" podUID="b595e184-27c8-4ffc-b50c-9764f013ef04" Nov 03 22:07:33.566746 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:33.566512 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk4s" podUID="5b28053b-cac3-4543-a09b-25915e6b3f0c" Nov 03 22:07:34.698459 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.698434 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeReady" Nov 03 22:07:34.698934 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.698559 2585 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 22:07:34.813748 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.813718 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5ztpx"] Nov 03 22:07:34.815668 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.815653 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rz85g"] Nov 03 22:07:34.815785 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.815767 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:34.817355 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.817335 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pjmrm"] Nov 03 22:07:34.817537 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.817443 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:07:34.818968 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.818949 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:34.866635 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.866612 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rz85g"] Nov 03 22:07:34.869732 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.869710 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:07:34.869879 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.869747 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:07:34.869879 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.869763 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsv4s\"" Nov 03 22:07:34.870063 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.869900 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:07:34.870535 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.870510 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5ztpx"] Nov 03 22:07:34.879157 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.879132 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pjmrm"] Nov 03 22:07:34.938620 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938590 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6d3f43-3683-4c19-8b29-849e581ccd37-cert\") pod \"ingress-canary-rz85g\" (UID: \"9e6d3f43-3683-4c19-8b29-849e581ccd37\") " pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:07:34.938744 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938634 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/790226ec-d17d-47e7-b32f-da97c9e983c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:34.938744 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938653 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:34.938744 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938724 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c056927-9f67-4a21-849b-dd71c606f245-config-volume\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:34.938842 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938767 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/790226ec-d17d-47e7-b32f-da97c9e983c2-data-volume\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:34.938842 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938785 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c056927-9f67-4a21-849b-dd71c606f245-metrics-tls\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:34.938842 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938804 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c056927-9f67-4a21-849b-dd71c606f245-tmp-dir\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:34.938918 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938839 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz5kc\" (UniqueName: \"kubernetes.io/projected/9e6d3f43-3683-4c19-8b29-849e581ccd37-kube-api-access-kz5kc\") pod \"ingress-canary-rz85g\" (UID: \"9e6d3f43-3683-4c19-8b29-849e581ccd37\") " pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:07:34.938918 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938910 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/790226ec-d17d-47e7-b32f-da97c9e983c2-crio-socket\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:34.938976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938927 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpsm4\" (UniqueName: \"kubernetes.io/projected/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-api-access-bpsm4\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:34.938976 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.938943 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzspk\" (UniqueName: \"kubernetes.io/projected/1c056927-9f67-4a21-849b-dd71c606f245-kube-api-access-xzspk\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:34.970030 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.969958 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:07:34.970030 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.969975 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:07:34.970196 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.970034 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:07:34.970196 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.970064 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:07:34.970196 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.970158 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2d6mp\"" Nov 03 22:07:34.970196 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.970159 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:07:34.970196 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.970179 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:07:34.970356 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:34.970227 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbwbg\"" Nov 03 22:07:35.040255 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040230 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/790226ec-d17d-47e7-b32f-da97c9e983c2-data-volume\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.040255 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040258 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c056927-9f67-4a21-849b-dd71c606f245-metrics-tls\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:35.040439 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040280 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c056927-9f67-4a21-849b-dd71c606f245-tmp-dir\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:35.040439 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040298 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5kc\" (UniqueName: \"kubernetes.io/projected/9e6d3f43-3683-4c19-8b29-849e581ccd37-kube-api-access-kz5kc\") pod \"ingress-canary-rz85g\" (UID: \"9e6d3f43-3683-4c19-8b29-849e581ccd37\") " pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:07:35.040439 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/790226ec-d17d-47e7-b32f-da97c9e983c2-crio-socket\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.040588 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040439 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpsm4\" (UniqueName: \"kubernetes.io/projected/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-api-access-bpsm4\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.040588 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzspk\" (UniqueName: \"kubernetes.io/projected/1c056927-9f67-4a21-849b-dd71c606f245-kube-api-access-xzspk\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:35.040674 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040624 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6d3f43-3683-4c19-8b29-849e581ccd37-cert\") pod \"ingress-canary-rz85g\" (UID: \"9e6d3f43-3683-4c19-8b29-849e581ccd37\") " pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:07:35.040674 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040635 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/790226ec-d17d-47e7-b32f-da97c9e983c2-data-volume\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.040768 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040692 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/790226ec-d17d-47e7-b32f-da97c9e983c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.040768 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040720 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.040768 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040731 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/790226ec-d17d-47e7-b32f-da97c9e983c2-crio-socket\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.040892 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.040854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c056927-9f67-4a21-849b-dd71c606f245-config-volume\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:35.041092 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.041068 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c056927-9f67-4a21-849b-dd71c606f245-tmp-dir\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:35.041379 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.041351 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.041803 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.041778 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c056927-9f67-4a21-849b-dd71c606f245-config-volume\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:35.044267 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.044243 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c056927-9f67-4a21-849b-dd71c606f245-metrics-tls\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:35.044342 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.044245 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6d3f43-3683-4c19-8b29-849e581ccd37-cert\") pod \"ingress-canary-rz85g\" (UID: \"9e6d3f43-3683-4c19-8b29-849e581ccd37\") " pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:07:35.044342 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.044327 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/790226ec-d17d-47e7-b32f-da97c9e983c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.097595 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.097566 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpsm4\" (UniqueName: \"kubernetes.io/projected/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-api-access-bpsm4\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.097717 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.097602 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5kc\" (UniqueName: \"kubernetes.io/projected/9e6d3f43-3683-4c19-8b29-849e581ccd37-kube-api-access-kz5kc\") pod \"ingress-canary-rz85g\" (UID: \"9e6d3f43-3683-4c19-8b29-849e581ccd37\") " pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:07:35.097717 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.097612 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzspk\" (UniqueName: \"kubernetes.io/projected/1c056927-9f67-4a21-849b-dd71c606f245-kube-api-access-xzspk\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:35.127438 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.127415 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:35.132384 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.132363 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:07:35.139639 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.139612 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:07:35.140550 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.140530 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj"] Nov 03 22:07:35.142198 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.142185 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" Nov 03 22:07:35.194813 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.194639 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj"] Nov 03 22:07:35.198564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.195897 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 22:07:35.198564 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.196032 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pkt2f\"" Nov 03 22:07:35.242439 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.242365 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9d22708-0a5b-41de-bd34-78e11212e865-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7zgbj\" (UID: \"c9d22708-0a5b-41de-bd34-78e11212e865\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" Nov 03 22:07:35.343149 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.343122 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9d22708-0a5b-41de-bd34-78e11212e865-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7zgbj\" (UID: \"c9d22708-0a5b-41de-bd34-78e11212e865\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" Nov 03 22:07:35.345448 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.345428 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9d22708-0a5b-41de-bd34-78e11212e865-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7zgbj\" (UID: \"c9d22708-0a5b-41de-bd34-78e11212e865\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" Nov 03 22:07:35.456288 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.456252 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" Nov 03 22:07:35.514800 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.514770 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rz85g"] Nov 03 22:07:35.515174 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:35.515144 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6d3f43_3683_4c19_8b29_849e581ccd37.slice/crio-f39aa78f21f42f418fb637667e7e4d006f900710474a682e9d6c43753bb1c552 WatchSource:0}: Error finding container f39aa78f21f42f418fb637667e7e4d006f900710474a682e9d6c43753bb1c552: Status 404 returned error can't find the container with id f39aa78f21f42f418fb637667e7e4d006f900710474a682e9d6c43753bb1c552 Nov 03 22:07:35.557955 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:35.557927 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c056927_9f67_4a21_849b_dd71c606f245.slice/crio-9e0e7cbae6e8494573445cbdb079506a5bef9b6162dd8114a8e70fd96f9d482a WatchSource:0}: Error finding container 9e0e7cbae6e8494573445cbdb079506a5bef9b6162dd8114a8e70fd96f9d482a: Status 404 returned error can't find the container with id 9e0e7cbae6e8494573445cbdb079506a5bef9b6162dd8114a8e70fd96f9d482a Nov 03 22:07:35.559123 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.559098 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5ztpx"] Nov 03 22:07:35.560733 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:35.560711 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790226ec_d17d_47e7_b32f_da97c9e983c2.slice/crio-783772461ca1e801e5bb09b48acf897b15f24cc68d727f4339a3c24255a50f44 WatchSource:0}: Error finding container 783772461ca1e801e5bb09b48acf897b15f24cc68d727f4339a3c24255a50f44: Status 404 returned error can't find the container with id 783772461ca1e801e5bb09b48acf897b15f24cc68d727f4339a3c24255a50f44 Nov 03 22:07:35.561896 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.561873 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pjmrm"] Nov 03 22:07:35.566690 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.566669 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:35.566759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.566694 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:35.567165 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.566895 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:35.619746 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.619719 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:07:35.619853 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.619791 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:07:35.619853 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.619821 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:07:35.620011 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.619992 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ng2g2\"" Nov 03 22:07:35.620046 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.620026 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxpq\"" Nov 03 22:07:35.620046 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.620031 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:07:35.620122 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.620108 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g6ff6\"" Nov 03 22:07:35.803296 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.803215 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjmrm" event={"ID":"790226ec-d17d-47e7-b32f-da97c9e983c2","Type":"ContainerStarted","Data":"f64074fe68ae53979b3a9b73576d959b02608b587f20fd49631296fa647c0848"} Nov 03 22:07:35.803296 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.803288 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjmrm" event={"ID":"790226ec-d17d-47e7-b32f-da97c9e983c2","Type":"ContainerStarted","Data":"783772461ca1e801e5bb09b48acf897b15f24cc68d727f4339a3c24255a50f44"} Nov 03 22:07:35.804374 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.804345 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rz85g" event={"ID":"9e6d3f43-3683-4c19-8b29-849e581ccd37","Type":"ContainerStarted","Data":"f39aa78f21f42f418fb637667e7e4d006f900710474a682e9d6c43753bb1c552"} Nov 03 22:07:35.805301 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.805274 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ztpx" event={"ID":"1c056927-9f67-4a21-849b-dd71c606f245","Type":"ContainerStarted","Data":"9e0e7cbae6e8494573445cbdb079506a5bef9b6162dd8114a8e70fd96f9d482a"} Nov 03 22:07:35.813139 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:35.813118 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d22708_0a5b_41de_bd34_78e11212e865.slice/crio-6ebc4734e6a8ed36070db7a8972f7749e3c517d9c80df80d2e7f82c1585dfc4d WatchSource:0}: Error finding container 6ebc4734e6a8ed36070db7a8972f7749e3c517d9c80df80d2e7f82c1585dfc4d: Status 404 returned error can't find the container with id 6ebc4734e6a8ed36070db7a8972f7749e3c517d9c80df80d2e7f82c1585dfc4d Nov 03 22:07:35.813358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:35.813270 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj"] Nov 03 22:07:36.809578 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:36.809533 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" event={"ID":"c9d22708-0a5b-41de-bd34-78e11212e865","Type":"ContainerStarted","Data":"6ebc4734e6a8ed36070db7a8972f7749e3c517d9c80df80d2e7f82c1585dfc4d"} Nov 03 22:07:37.818432 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:37.818350 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjmrm" event={"ID":"790226ec-d17d-47e7-b32f-da97c9e983c2","Type":"ContainerStarted","Data":"0ab7d2c2af395f176a61ae94b0f8bdd40cce21c39f5399922cce99c61af2c477"} Nov 03 22:07:37.820077 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:37.820042 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" event={"ID":"c9d22708-0a5b-41de-bd34-78e11212e865","Type":"ContainerStarted","Data":"924f9377789a4cde933d85779d21555ee1a13eaf2a3a24cd4a89c496d7f55b3c"} Nov 03 22:07:37.820303 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:37.820275 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" Nov 03 22:07:37.826358 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:37.826333 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" Nov 03 22:07:37.933507 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:37.933449 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" podStartSLOduration=1.249363254 podStartE2EDuration="2.93343776s" totalImagesPullingTime="1.684074506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:37.933047476 +0000 UTC m=+51.012496511" watchObservedRunningTime="2026-11-03 22:07:37.93343776 +0000 UTC m=+51.012886787" Nov 03 22:07:38.787075 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.787036 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wkmds"] Nov 03 22:07:38.789586 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.789565 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:38.823856 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.823820 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ztpx" event={"ID":"1c056927-9f67-4a21-849b-dd71c606f245","Type":"ContainerStarted","Data":"43906c1532f922f14b361a29d5108b8d4571960d0c280a4a8ee2abe40cd2a759"} Nov 03 22:07:38.824238 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.823870 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ztpx" event={"ID":"1c056927-9f67-4a21-849b-dd71c606f245","Type":"ContainerStarted","Data":"fd9987bf96f997ed204f6f5b6b2f107a733ce574002b16236c375c8ad23d4a9b"} Nov 03 22:07:38.893452 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.893413 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:07:38.893452 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.893439 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 22:07:38.893601 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.893513 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:07:38.893601 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.893543 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 22:07:38.894195 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.894177 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wkmds"] Nov 03 22:07:38.943925 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.943905 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:07:38.944068 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.944053 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nq6b6\"" Nov 03 22:07:38.971558 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.971534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f26ebc-df35-404e-9466-9f5d73478ef7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:38.971643 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.971560 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cpph\" (UniqueName: \"kubernetes.io/projected/a0f26ebc-df35-404e-9466-9f5d73478ef7-kube-api-access-2cpph\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:38.971643 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.971595 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:38.971718 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:38.971669 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:39.067840 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.067791 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5ztpx" podStartSLOduration=2.7564932989999997 podStartE2EDuration="5.067779018s" totalImagesPullingTime="2.311285719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:39.066908917 +0000 UTC m=+52.146357954" watchObservedRunningTime="2026-11-03 22:07:39.067779018 +0000 UTC m=+52.147228054" Nov 03 22:07:39.072608 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.072582 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:39.072701 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.072626 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:39.072754 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.072701 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f26ebc-df35-404e-9466-9f5d73478ef7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:39.072754 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.072730 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpph\" (UniqueName: \"kubernetes.io/projected/a0f26ebc-df35-404e-9466-9f5d73478ef7-kube-api-access-2cpph\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:39.073375 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.073353 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f26ebc-df35-404e-9466-9f5d73478ef7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:39.075015 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.074987 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:39.075132 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.075113 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:39.129467 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.129434 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpph\" (UniqueName: \"kubernetes.io/projected/a0f26ebc-df35-404e-9466-9f5d73478ef7-kube-api-access-2cpph\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:39.360519 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:39.360401 2585 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage546825760/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261103T220737Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=598d49446376374d7ec41b83a0cc1765396f9deeb66cf2bf7fa2bf36a3d83343®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793744557~hmac=b2dea0c25906bfd4476f8863594ea36c1f311add60ea3bb5cbe2f376cb0b9e58\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb08b4955abf8a0ffea0e4f2e446fbb6a5f3271312a996633901621b91b106ab" Nov 03 22:07:39.360678 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:39.360653 2585 kuberuntime_manager.go:1707] "Unhandled Error" err="container serve-healthcheck-canary start failed in pod ingress-canary-rz85g_openshift-ingress-canary(9e6d3f43-3683-4c19-8b29-849e581ccd37): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage546825760/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261103T220737Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=598d49446376374d7ec41b83a0cc1765396f9deeb66cf2bf7fa2bf36a3d83343®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793744557~hmac=b2dea0c25906bfd4476f8863594ea36c1f311add60ea3bb5cbe2f376cb0b9e58\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" logger="UnhandledError" Nov 03 22:07:39.360731 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:39.360703 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"serve-healthcheck-canary\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage546825760/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261103T220737Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=598d49446376374d7ec41b83a0cc1765396f9deeb66cf2bf7fa2bf36a3d83343®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793744557~hmac=b2dea0c25906bfd4476f8863594ea36c1f311add60ea3bb5cbe2f376cb0b9e58\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress-canary/ingress-canary-rz85g" podUID="9e6d3f43-3683-4c19-8b29-849e581ccd37" Nov 03 22:07:39.402821 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.402789 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:07:39.762798 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.762769 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wkmds"] Nov 03 22:07:39.792365 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:39.792337 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f26ebc_df35_404e_9466_9f5d73478ef7.slice/crio-a3230941b68331a7751394a36ddecba182890207f963e234e8ac5ca1399c038d WatchSource:0}: Error finding container a3230941b68331a7751394a36ddecba182890207f963e234e8ac5ca1399c038d: Status 404 returned error can't find the container with id a3230941b68331a7751394a36ddecba182890207f963e234e8ac5ca1399c038d Nov 03 22:07:39.827230 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.827206 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" event={"ID":"a0f26ebc-df35-404e-9466-9f5d73478ef7","Type":"ContainerStarted","Data":"a3230941b68331a7751394a36ddecba182890207f963e234e8ac5ca1399c038d"} Nov 03 22:07:39.829270 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.829248 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjmrm" event={"ID":"790226ec-d17d-47e7-b32f-da97c9e983c2","Type":"ContainerStarted","Data":"c455b2f14d0395420ea350e2026de02ba93443554955fe41bf3746153c5b6140"} Nov 03 22:07:39.829740 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:39.829721 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:39.830930 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:39.830891 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"serve-healthcheck-canary\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb08b4955abf8a0ffea0e4f2e446fbb6a5f3271312a996633901621b91b106ab\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage546825760/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261103T220737Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=598d49446376374d7ec41b83a0cc1765396f9deeb66cf2bf7fa2bf36a3d83343®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793744557~hmac=b2dea0c25906bfd4476f8863594ea36c1f311add60ea3bb5cbe2f376cb0b9e58\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress-canary/ingress-canary-rz85g" podUID="9e6d3f43-3683-4c19-8b29-849e581ccd37" Nov 03 22:07:40.078855 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:40.078756 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pjmrm" podStartSLOduration=2.992729317 podStartE2EDuration="6.078741487s" totalImagesPullingTime="3.08601217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:07:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:40.077839749 +0000 UTC m=+53.157288796" watchObservedRunningTime="2026-11-03 22:07:40.078741487 +0000 UTC m=+53.158190522" Nov 03 22:07:41.837396 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:41.837357 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" event={"ID":"a0f26ebc-df35-404e-9466-9f5d73478ef7","Type":"ContainerStarted","Data":"6406ffd2949b797a04d8650de3572f7159971626f823e2ab0ef6433d850cd828"} Nov 03 22:07:41.837772 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:41.837427 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" event={"ID":"a0f26ebc-df35-404e-9466-9f5d73478ef7","Type":"ContainerStarted","Data":"08732a4aab6902e93ab4674b29545ab993807a2cf1d843b007347e9eaf1b87c8"} Nov 03 22:07:41.971966 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:41.971917 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" podStartSLOduration=2.125082688 podStartE2EDuration="3.971904263s" totalImagesPullingTime="1.846821575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:41.971439197 +0000 UTC m=+55.050888244" watchObservedRunningTime="2026-11-03 22:07:41.971904263 +0000 UTC m=+55.051353309" Nov 03 22:07:42.765261 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:42.765235 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:07:44.089663 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.089631 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-64cffc769c-fpv4l"] Nov 03 22:07:44.092070 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.092045 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.146221 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.146187 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 22:07:44.146394 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.146293 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 22:07:44.196534 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.196435 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 22:07:44.196707 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.196684 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lr7tn\"" Nov 03 22:07:44.197541 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.197517 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cffc769c-fpv4l"] Nov 03 22:07:44.210100 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.210076 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.210211 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.210128 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.210211 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.210146 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.210320 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.210213 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.210373 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.210341 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-tmp\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.210423 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.210400 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.210484 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.210465 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.210552 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.210522 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.300341 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.300157 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 22:07:44.300341 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.300298 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 22:07:44.300687 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.300651 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 22:07:44.301101 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.300852 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 22:07:44.301101 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.300949 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 22:07:44.311992 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.311813 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-tmp\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.311992 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.311880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.311992 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.311909 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.311992 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.311931 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.311992 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.311971 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.312291 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.312273 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-tmp\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.312531 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.312482 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.312637 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.312547 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.312637 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.312577 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.312742 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.312659 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.312742 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.312663 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.312914 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.312888 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.313552 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.313529 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.315045 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.315023 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.315267 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.315242 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.372785 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.372724 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.401608 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.401582 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:44.767546 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.767517 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cffc769c-fpv4l"] Nov 03 22:07:44.796355 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:44.796323 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e37b4fe_1eec_4bfe_88ee_95cf34d72e48.slice/crio-4422470e83279867e1ebbb50306c4393786557de2d6685d16207563890000f1b WatchSource:0}: Error finding container 4422470e83279867e1ebbb50306c4393786557de2d6685d16207563890000f1b: Status 404 returned error can't find the container with id 4422470e83279867e1ebbb50306c4393786557de2d6685d16207563890000f1b Nov 03 22:07:44.849083 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:44.849044 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cffc769c-fpv4l" event={"ID":"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48","Type":"ContainerStarted","Data":"4422470e83279867e1ebbb50306c4393786557de2d6685d16207563890000f1b"} Nov 03 22:07:45.338981 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.338949 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db"] Nov 03 22:07:45.341652 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.341626 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.395692 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.395666 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 22:07:45.395881 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.395863 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mtmht\"" Nov 03 22:07:45.395983 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.395962 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db"] Nov 03 22:07:45.396038 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.395987 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 22:07:45.420308 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.420280 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.420387 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.420333 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64bmn\" (UniqueName: \"kubernetes.io/projected/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-kube-api-access-64bmn\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.420387 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.420373 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.420465 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.420389 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.457371 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.457346 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n"] Nov 03 22:07:45.459443 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.459428 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.469173 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.469152 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wbfjj"] Nov 03 22:07:45.470967 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.470953 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.515833 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.515811 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n"] Nov 03 22:07:45.520298 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.520273 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 22:07:45.520298 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.520278 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mrtjf\"" Nov 03 22:07:45.520444 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.520353 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 22:07:45.520444 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.520385 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 22:07:45.520827 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.520809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.520859 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.520842 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-textfile\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.520893 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.520862 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2lc\" (UniqueName: \"kubernetes.io/projected/02608c76-a776-4f06-94d3-eb3527a9dcdf-kube-api-access-fg2lc\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.520927 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.520896 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-accelerators-collector-config\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.521028 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521017 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.521069 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521054 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-sys\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.521098 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521085 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnp77\" (UniqueName: \"kubernetes.io/projected/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-api-access-nnp77\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.521129 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521103 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-root\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.521129 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521119 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-wtmp\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.521184 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521149 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.521214 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521187 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.521248 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521232 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-metrics-client-ca\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.521354 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521333 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/433b8aa2-23d3-432b-abf6-15b7b334b287-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.521460 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521362 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.521460 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521383 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.521460 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521406 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.521641 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521602 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64bmn\" (UniqueName: \"kubernetes.io/projected/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-kube-api-access-64bmn\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.521693 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521645 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-tls\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.521693 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521682 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.521801 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.521779 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.523449 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.523426 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.523812 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.523788 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.523885 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.523800 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:07:45.523885 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.523832 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:07:45.523885 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.523839 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:07:45.524023 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.523908 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xzmgl\"" Nov 03 22:07:45.580452 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.580431 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64bmn\" (UniqueName: \"kubernetes.io/projected/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-kube-api-access-64bmn\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.622203 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622138 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnp77\" (UniqueName: \"kubernetes.io/projected/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-api-access-nnp77\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.622203 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622168 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-root\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622203 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622185 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-wtmp\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622432 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622261 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-root\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622432 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622289 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-wtmp\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622432 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622321 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.622432 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622375 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-metrics-client-ca\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622662 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622435 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/433b8aa2-23d3-432b-abf6-15b7b334b287-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.622662 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622466 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.622662 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622513 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622662 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-tls\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622662 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622594 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.622662 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622643 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-textfile\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622933 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622669 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2lc\" (UniqueName: \"kubernetes.io/projected/02608c76-a776-4f06-94d3-eb3527a9dcdf-kube-api-access-fg2lc\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622933 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622711 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-accelerators-collector-config\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622933 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622740 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.622933 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622773 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-sys\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.622933 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622816 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/433b8aa2-23d3-432b-abf6-15b7b334b287-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.622933 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.622858 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-sys\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.623208 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.623073 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-metrics-client-ca\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.623441 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.623376 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-textfile\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.623441 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.623394 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-accelerators-collector-config\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.623675 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.623639 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.623782 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.623723 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.624885 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.624861 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.624996 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.624976 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.625118 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.625099 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.625203 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.625187 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-tls\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.653913 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.653891 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:07:45.679233 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.679212 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2lc\" (UniqueName: \"kubernetes.io/projected/02608c76-a776-4f06-94d3-eb3527a9dcdf-kube-api-access-fg2lc\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.679515 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.679485 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnp77\" (UniqueName: \"kubernetes.io/projected/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-api-access-nnp77\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.770854 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.770822 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:07:45.781700 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.781674 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:07:45.793790 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:45.793759 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02608c76_a776_4f06_94d3_eb3527a9dcdf.slice/crio-48902d27bbc64df82c086b84be533621916319b412b2f122b5ae537d542f60af WatchSource:0}: Error finding container 48902d27bbc64df82c086b84be533621916319b412b2f122b5ae537d542f60af: Status 404 returned error can't find the container with id 48902d27bbc64df82c086b84be533621916319b412b2f122b5ae537d542f60af Nov 03 22:07:45.853553 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:45.853525 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wbfjj" event={"ID":"02608c76-a776-4f06-94d3-eb3527a9dcdf","Type":"ContainerStarted","Data":"48902d27bbc64df82c086b84be533621916319b412b2f122b5ae537d542f60af"} Nov 03 22:07:46.067458 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:46.067430 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded75a83a_9f07_4a2f_be7f_1a0b249728c0.slice/crio-450d028018fda0a97df38ed64931cf3a2bf63132fb0809cc40dcf1db702f8278 WatchSource:0}: Error finding container 450d028018fda0a97df38ed64931cf3a2bf63132fb0809cc40dcf1db702f8278: Status 404 returned error can't find the container with id 450d028018fda0a97df38ed64931cf3a2bf63132fb0809cc40dcf1db702f8278 Nov 03 22:07:46.067579 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:46.067461 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db"] Nov 03 22:07:46.162663 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:46.162635 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n"] Nov 03 22:07:46.858840 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:46.858798 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" event={"ID":"433b8aa2-23d3-432b-abf6-15b7b334b287","Type":"ContainerStarted","Data":"4a9054cd3149355ebd80f651406354a3552dea2bbfbbcd579869bf7e3c819c08"} Nov 03 22:07:46.861101 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:46.861051 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" event={"ID":"ed75a83a-9f07-4a2f-be7f-1a0b249728c0","Type":"ContainerStarted","Data":"eecabc209a6d8ec02e4e24bb975a83d46edd54a40d14077978e5363636def46c"} Nov 03 22:07:46.861232 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:46.861135 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" event={"ID":"ed75a83a-9f07-4a2f-be7f-1a0b249728c0","Type":"ContainerStarted","Data":"a1bb370584b5ad4db0ce0d3f853f9b2ce4764615314d1ad3951455ac33434741"} Nov 03 22:07:46.861232 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:46.861165 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" event={"ID":"ed75a83a-9f07-4a2f-be7f-1a0b249728c0","Type":"ContainerStarted","Data":"450d028018fda0a97df38ed64931cf3a2bf63132fb0809cc40dcf1db702f8278"} Nov 03 22:07:47.593115 ip-10-0-134-57 kubenswrapper[2585]: E1103 22:07:47.592229 2585 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02608c76_a776_4f06_94d3_eb3527a9dcdf.slice/crio-conmon-b2b80ced960f71cda509e92d6f1b85a8f4ffc508e1c2c2a4c073688a16bc4369.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:07:47.868242 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:47.867570 2585 generic.go:356] "Generic (PLEG): container finished" podID="02608c76-a776-4f06-94d3-eb3527a9dcdf" containerID="b2b80ced960f71cda509e92d6f1b85a8f4ffc508e1c2c2a4c073688a16bc4369" exitCode=0 Nov 03 22:07:47.868242 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:47.867647 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wbfjj" event={"ID":"02608c76-a776-4f06-94d3-eb3527a9dcdf","Type":"ContainerDied","Data":"b2b80ced960f71cda509e92d6f1b85a8f4ffc508e1c2c2a4c073688a16bc4369"} Nov 03 22:07:48.037532 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.036650 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8ddd84555-gsxvd"] Nov 03 22:07:48.040906 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.040879 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.067662 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.067634 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8ddd84555-gsxvd"] Nov 03 22:07:48.094418 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.094385 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 22:07:48.094579 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.094442 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 22:07:48.094579 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.094478 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 22:07:48.094700 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.094579 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4ri5bql2iielr\"" Nov 03 22:07:48.094700 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.094697 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2s7p7\"" Nov 03 22:07:48.094798 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.094736 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 22:07:48.094849 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.094804 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 22:07:48.145172 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.145102 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-metrics-client-ca\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.145334 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.145170 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.145334 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.145284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.145334 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.145321 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.145745 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.145372 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.145745 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.145419 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-grpc-tls\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.145745 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.145454 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-tmp\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.145745 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.145481 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-tls\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.145745 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.145528 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nrm4\" (UniqueName: \"kubernetes.io/projected/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-kube-api-access-9nrm4\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.248759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.246544 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-grpc-tls\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.248759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.246602 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-tmp\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.248759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.246630 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-tls\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.248759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.246656 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrm4\" (UniqueName: \"kubernetes.io/projected/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-kube-api-access-9nrm4\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.248759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.246699 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-metrics-client-ca\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.248759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.246763 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.248759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.246809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.248759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.246837 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.248759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.246886 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.248759 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.248124 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-tmp\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.249386 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.248889 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-metrics-client-ca\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.250688 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.250648 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-tls\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.251285 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.251258 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.251740 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.251707 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-grpc-tls\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.253717 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.253699 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.253832 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.253810 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.256773 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.256531 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.304384 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.304358 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrm4\" (UniqueName: \"kubernetes.io/projected/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-kube-api-access-9nrm4\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.360771 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.360746 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:48.838458 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.838430 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8ddd84555-gsxvd"] Nov 03 22:07:48.873147 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.873109 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cffc769c-fpv4l" event={"ID":"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48","Type":"ContainerStarted","Data":"2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537"} Nov 03 22:07:48.875431 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.875398 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wbfjj" event={"ID":"02608c76-a776-4f06-94d3-eb3527a9dcdf","Type":"ContainerStarted","Data":"04225f4ce5e30ae330eb6e777f80091dba74c0062089c9d9e5468b50f2749cdc"} Nov 03 22:07:48.875570 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.875459 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wbfjj" event={"ID":"02608c76-a776-4f06-94d3-eb3527a9dcdf","Type":"ContainerStarted","Data":"a6fe8b9e867e87fbdd4142dbcf1c9ec393d7a6e2463e5eee73af76f33ad75bb0"} Nov 03 22:07:48.878411 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.878382 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" event={"ID":"433b8aa2-23d3-432b-abf6-15b7b334b287","Type":"ContainerStarted","Data":"3380ad7b4f6072975bdbdba463c5ec414467221342789f9c72ae25d7e848f9bb"} Nov 03 22:07:48.878530 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.878439 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" event={"ID":"433b8aa2-23d3-432b-abf6-15b7b334b287","Type":"ContainerStarted","Data":"3717b9375c1b34126c9307c40d5f5fe68eb88b3f5504bbb71e0c248c3de68400"} Nov 03 22:07:48.878530 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.878466 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" event={"ID":"433b8aa2-23d3-432b-abf6-15b7b334b287","Type":"ContainerStarted","Data":"d16d539dfd2b3936b02351517ac94de3190b124e8fde6b57c5a22e817a4d30bb"} Nov 03 22:07:48.880816 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.880787 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" event={"ID":"ed75a83a-9f07-4a2f-be7f-1a0b249728c0","Type":"ContainerStarted","Data":"566737441c8cbcc32122bf5831efb52949bc6d034c322128e3d1b1b6ef5098e4"} Nov 03 22:07:48.884544 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:48.884520 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd962bd32_3fc5_4a46_ba44_0d08701a6cd0.slice/crio-6dda3470346bc8c16e5fdabb3a57d903822ac594ecc56d23e314e205703fa3b8 WatchSource:0}: Error finding container 6dda3470346bc8c16e5fdabb3a57d903822ac594ecc56d23e314e205703fa3b8: Status 404 returned error can't find the container with id 6dda3470346bc8c16e5fdabb3a57d903822ac594ecc56d23e314e205703fa3b8 Nov 03 22:07:48.991954 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:48.991904 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-64cffc769c-fpv4l" podStartSLOduration=1.45066845 podStartE2EDuration="4.991890786s" totalImagesPullingTime="3.541222336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:48.99140799 +0000 UTC m=+62.070857027" watchObservedRunningTime="2026-11-03 22:07:48.991890786 +0000 UTC m=+62.071339821" Nov 03 22:07:49.110387 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:49.110284 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" podStartSLOduration=2.124556423 podStartE2EDuration="4.110271138s" totalImagesPullingTime="1.985714715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:49.109544516 +0000 UTC m=+62.188993552" watchObservedRunningTime="2026-11-03 22:07:49.110271138 +0000 UTC m=+62.189720169" Nov 03 22:07:49.225163 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:49.225106 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" podStartSLOduration=2.488476036 podStartE2EDuration="4.225091023s" totalImagesPullingTime="1.736614987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:49.22425096 +0000 UTC m=+62.303699997" watchObservedRunningTime="2026-11-03 22:07:49.225091023 +0000 UTC m=+62.304540059" Nov 03 22:07:49.342385 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:49.342336 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wbfjj" podStartSLOduration=2.674566905 podStartE2EDuration="4.342321782s" totalImagesPullingTime="1.583525661s" totalInitContainerRuntime="84.229216ms" isStatefulPod=true podCreationTimestamp="2026-11-03 22:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:49.340818944 +0000 UTC m=+62.420267981" watchObservedRunningTime="2026-11-03 22:07:49.342321782 +0000 UTC m=+62.421770829" Nov 03 22:07:49.834505 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:49.834463 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5ztpx" Nov 03 22:07:49.885410 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:49.885375 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" event={"ID":"d962bd32-3fc5-4a46-ba44-0d08701a6cd0","Type":"ContainerStarted","Data":"6dda3470346bc8c16e5fdabb3a57d903822ac594ecc56d23e314e205703fa3b8"} Nov 03 22:07:50.360137 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.360092 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-779ff6cb48-8rzf9"] Nov 03 22:07:50.362593 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.362566 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.416081 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.416051 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 22:07:50.417733 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.417706 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 22:07:50.417884 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.417765 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 22:07:50.417884 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.417779 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2hth5\"" Nov 03 22:07:50.417884 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.417819 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 22:07:50.417884 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.417857 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1o2rm9ggtu00i\"" Nov 03 22:07:50.418706 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.418690 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-779ff6cb48-8rzf9"] Nov 03 22:07:50.473543 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.473488 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-client-certs\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.473684 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.473552 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5692356c-6609-46d4-9cd7-513a0d883779-audit-log\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.473684 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.473670 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npwx8\" (UniqueName: \"kubernetes.io/projected/5692356c-6609-46d4-9cd7-513a0d883779-kube-api-access-npwx8\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.473773 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.473753 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-client-ca-bundle\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.473815 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.473773 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-metrics-server-audit-profiles\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.473815 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.473796 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.473904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.473815 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-tls\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.574876 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.574838 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-client-ca-bundle\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.575058 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.574892 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-metrics-server-audit-profiles\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.575058 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.574920 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.575058 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.574939 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-tls\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.575058 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.574994 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-client-certs\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.575058 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.575019 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5692356c-6609-46d4-9cd7-513a0d883779-audit-log\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.575321 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.575064 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npwx8\" (UniqueName: \"kubernetes.io/projected/5692356c-6609-46d4-9cd7-513a0d883779-kube-api-access-npwx8\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.575842 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.575761 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.575842 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.575768 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5692356c-6609-46d4-9cd7-513a0d883779-audit-log\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.576175 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.576150 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-metrics-server-audit-profiles\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.577794 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.577771 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-client-ca-bundle\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.578234 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.578216 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-tls\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.578337 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.578319 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-client-certs\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.635077 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.635006 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npwx8\" (UniqueName: \"kubernetes.io/projected/5692356c-6609-46d4-9cd7-513a0d883779-kube-api-access-npwx8\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:50.674251 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:50.674049 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:07:51.094986 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:51.094899 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5692356c_6609_46d4_9cd7_513a0d883779.slice/crio-9ea7d018178e39398cebb348d374c83658c88d49ed28fb15c7bde2480c853ab4 WatchSource:0}: Error finding container 9ea7d018178e39398cebb348d374c83658c88d49ed28fb15c7bde2480c853ab4: Status 404 returned error can't find the container with id 9ea7d018178e39398cebb348d374c83658c88d49ed28fb15c7bde2480c853ab4 Nov 03 22:07:51.096033 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.095956 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-779ff6cb48-8rzf9"] Nov 03 22:07:51.141134 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.141096 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f"] Nov 03 22:07:51.144531 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.144503 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.181215 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.181187 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.181353 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.181239 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.181353 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.181312 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scj68\" (UniqueName: \"kubernetes.io/projected/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-kube-api-access-scj68\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.181460 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.181358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-serving-certs-ca-bundle\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.181531 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.181470 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-federate-client-tls\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.181531 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.181521 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.181620 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.181584 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-client-tls\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.181666 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.181629 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-metrics-client-ca\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.201213 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.198789 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 22:07:51.201646 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.201616 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 22:07:51.204188 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.204168 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 22:07:51.248434 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.248374 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-khcr8\"" Nov 03 22:07:51.248623 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.248508 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 22:07:51.248623 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.248582 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 22:07:51.282451 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.282411 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scj68\" (UniqueName: \"kubernetes.io/projected/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-kube-api-access-scj68\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.282635 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.282460 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-serving-certs-ca-bundle\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.282635 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.282568 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-federate-client-tls\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.282635 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.282593 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.282635 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.282633 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-client-tls\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.282844 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.282660 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-metrics-client-ca\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.282844 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.282708 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.282844 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.282744 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.283534 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.283509 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-serving-certs-ca-bundle\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.283662 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.283490 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-metrics-client-ca\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.286208 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.286003 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.286208 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.286119 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-federate-client-tls\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.286208 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.286181 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-client-tls\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.286344 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.286225 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.298472 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.298447 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 22:07:51.300122 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.300102 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f"] Nov 03 22:07:51.303990 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.303970 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.339845 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.339822 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scj68\" (UniqueName: \"kubernetes.io/projected/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-kube-api-access-scj68\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.460384 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.460348 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:07:51.885880 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:51.885852 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d71ea07_fa84_4b2a_b06d_f7b05d7a126d.slice/crio-4607c7bf1180bce2fff71e4197866c36518890c83577c0021a5e656a19b850be WatchSource:0}: Error finding container 4607c7bf1180bce2fff71e4197866c36518890c83577c0021a5e656a19b850be: Status 404 returned error can't find the container with id 4607c7bf1180bce2fff71e4197866c36518890c83577c0021a5e656a19b850be Nov 03 22:07:51.887249 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.887221 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f"] Nov 03 22:07:51.895092 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.895068 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" event={"ID":"d962bd32-3fc5-4a46-ba44-0d08701a6cd0","Type":"ContainerStarted","Data":"38ed132c460e0c63c9c3ea9bbb9d7b8d92989921ded3d7726e3d8ff9b29353f2"} Nov 03 22:07:51.895184 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.895126 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" event={"ID":"d962bd32-3fc5-4a46-ba44-0d08701a6cd0","Type":"ContainerStarted","Data":"a6ab548fbed9099cf4a77d695de77c6897a936cfe513158891e581807aa261dc"} Nov 03 22:07:51.895184 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.895154 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" event={"ID":"d962bd32-3fc5-4a46-ba44-0d08701a6cd0","Type":"ContainerStarted","Data":"1ec43b00b31f1e160d0dcf26b54a00953e4cb76dafb47694fd3b2b16b64abd6f"} Nov 03 22:07:51.896163 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.896144 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" event={"ID":"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d","Type":"ContainerStarted","Data":"4607c7bf1180bce2fff71e4197866c36518890c83577c0021a5e656a19b850be"} Nov 03 22:07:51.897218 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:51.897198 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" event={"ID":"5692356c-6609-46d4-9cd7-513a0d883779","Type":"ContainerStarted","Data":"9ea7d018178e39398cebb348d374c83658c88d49ed28fb15c7bde2480c853ab4"} Nov 03 22:07:53.200989 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.200961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:53.257517 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.253774 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:07:53.268547 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.265159 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:53.303653 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.303179 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:53.356313 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.356278 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:07:53.365754 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.365722 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:53.390136 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.390089 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxpq\"" Nov 03 22:07:53.398211 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.398184 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:07:53.404035 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.404004 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:53.457249 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.457223 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:07:53.516803 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.516775 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:07:53.528341 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.528310 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:53.683480 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.683458 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ng2g2\"" Nov 03 22:07:53.692665 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.692627 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:53.709049 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.709023 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g6ff6\"" Nov 03 22:07:53.718183 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.718149 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:07:53.775904 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.775375 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkldp"] Nov 03 22:07:53.777743 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:53.777701 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b9a551_4faa_46fb_8d85_9a39e39d9191.slice/crio-4a118167180796978d0e8297a20c2a4554609e8740f3e3f4f2ebefbe142620c9 WatchSource:0}: Error finding container 4a118167180796978d0e8297a20c2a4554609e8740f3e3f4f2ebefbe142620c9: Status 404 returned error can't find the container with id 4a118167180796978d0e8297a20c2a4554609e8740f3e3f4f2ebefbe142620c9 Nov 03 22:07:53.909374 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.909335 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" event={"ID":"5692356c-6609-46d4-9cd7-513a0d883779","Type":"ContainerStarted","Data":"17bdf2f6039998e5a6da393ef730ed639a78249515b95cf7dde2fef56c01ca8a"} Nov 03 22:07:53.912843 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.912818 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" event={"ID":"d962bd32-3fc5-4a46-ba44-0d08701a6cd0","Type":"ContainerStarted","Data":"59137e51844f5fa8db28363a1200764b144c16612a02f22b4f87b25f6896d313"} Nov 03 22:07:53.912984 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.912878 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" event={"ID":"d962bd32-3fc5-4a46-ba44-0d08701a6cd0","Type":"ContainerStarted","Data":"fc36c4f5954f04c2bfcd3d097707ae64018a2217d2c643011c553799edae9e16"} Nov 03 22:07:53.912984 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.912909 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" event={"ID":"d962bd32-3fc5-4a46-ba44-0d08701a6cd0","Type":"ContainerStarted","Data":"5cc599dfc9edd1df51be70acd8bc19e9598b085f76ee09fd4a0e6452ea178944"} Nov 03 22:07:53.912984 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.912977 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:07:53.914145 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.914117 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkldp" event={"ID":"03b9a551-4faa-46fb-8d85-9a39e39d9191","Type":"ContainerStarted","Data":"4a118167180796978d0e8297a20c2a4554609e8740f3e3f4f2ebefbe142620c9"} Nov 03 22:07:53.975236 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:53.974830 2585 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cffc769c-fpv4l"] Nov 03 22:07:54.014899 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.014863 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-588668ffc6-29dl9"] Nov 03 22:07:54.017912 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.017892 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.033686 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.033642 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" podStartSLOduration=2.06863478 podStartE2EDuration="4.033625632s" totalImagesPullingTime="1.964990852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:54.032956448 +0000 UTC m=+67.112405483" watchObservedRunningTime="2026-11-03 22:07:54.033625632 +0000 UTC m=+67.113074668" Nov 03 22:07:54.070406 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:54.070378 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb595e184_27c8_4ffc_b50c_9764f013ef04.slice/crio-08ce5f5336a6ce6e86809d215d3b7b8c094af482d42dc87a6ff9941869177905 WatchSource:0}: Error finding container 08ce5f5336a6ce6e86809d215d3b7b8c094af482d42dc87a6ff9941869177905: Status 404 returned error can't find the container with id 08ce5f5336a6ce6e86809d215d3b7b8c094af482d42dc87a6ff9941869177905 Nov 03 22:07:54.071305 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.071280 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9bqcf"] Nov 03 22:07:54.079543 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.079485 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588668ffc6-29dl9"] Nov 03 22:07:54.100596 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:54.100563 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b28053b_cac3_4543_a09b_25915e6b3f0c.slice/crio-b584ea2c20c71c423c63f7de2148c394953c3292384e34d3caa883d5cab9013c WatchSource:0}: Error finding container b584ea2c20c71c423c63f7de2148c394953c3292384e34d3caa883d5cab9013c: Status 404 returned error can't find the container with id b584ea2c20c71c423c63f7de2148c394953c3292384e34d3caa883d5cab9013c Nov 03 22:07:54.101190 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.101168 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lwk4s"] Nov 03 22:07:54.109924 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.109897 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.110040 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.110001 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.110097 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.110044 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.110097 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.110075 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.110198 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.110108 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beccabec-c5c9-4f17-a8ba-c23e1254c391-tmp\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.110198 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.110140 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.110198 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.110166 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.110313 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.110233 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.153953 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.153894 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" podStartSLOduration=3.095093356 podStartE2EDuration="7.153874896s" totalImagesPullingTime="4.05878154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:54.152823465 +0000 UTC m=+67.232272502" watchObservedRunningTime="2026-11-03 22:07:54.153874896 +0000 UTC m=+67.233323936" Nov 03 22:07:54.211017 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.210979 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.211017 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.211023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.211568 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.211051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.211568 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.211082 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.211568 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.211226 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beccabec-c5c9-4f17-a8ba-c23e1254c391-tmp\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.211568 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.211278 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.211568 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.211312 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.211568 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.211344 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.211863 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.211639 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beccabec-c5c9-4f17-a8ba-c23e1254c391-tmp\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.212010 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.211976 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.212611 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.212576 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.212712 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.212643 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.212712 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.212694 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.214078 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.214053 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.214278 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.214253 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.273613 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.273546 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.329262 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.329239 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:07:54.402667 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.402636 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:07:54.698274 ip-10-0-134-57 kubenswrapper[2585]: W1103 22:07:54.698235 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeccabec_c5c9_4f17_a8ba_c23e1254c391.slice/crio-6820ffc16879a9154d682d03811d75c42159650624ba2f39cffe69f6302210c6 WatchSource:0}: Error finding container 6820ffc16879a9154d682d03811d75c42159650624ba2f39cffe69f6302210c6: Status 404 returned error can't find the container with id 6820ffc16879a9154d682d03811d75c42159650624ba2f39cffe69f6302210c6 Nov 03 22:07:54.698380 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.698284 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588668ffc6-29dl9"] Nov 03 22:07:54.919905 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.919825 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588668ffc6-29dl9" event={"ID":"beccabec-c5c9-4f17-a8ba-c23e1254c391","Type":"ContainerStarted","Data":"95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843"} Nov 03 22:07:54.919905 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.919890 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588668ffc6-29dl9" event={"ID":"beccabec-c5c9-4f17-a8ba-c23e1254c391","Type":"ContainerStarted","Data":"6820ffc16879a9154d682d03811d75c42159650624ba2f39cffe69f6302210c6"} Nov 03 22:07:54.922087 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.922058 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" event={"ID":"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d","Type":"ContainerStarted","Data":"80202c79467cc34fb4bf71dde625341dd5ebd1ba375e8e542a6960e7201ed5c6"} Nov 03 22:07:54.923464 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.923436 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lwk4s" event={"ID":"5b28053b-cac3-4543-a09b-25915e6b3f0c","Type":"ContainerStarted","Data":"b584ea2c20c71c423c63f7de2148c394953c3292384e34d3caa883d5cab9013c"} Nov 03 22:07:54.924905 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:54.924876 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9bqcf" event={"ID":"b595e184-27c8-4ffc-b50c-9764f013ef04","Type":"ContainerStarted","Data":"08ce5f5336a6ce6e86809d215d3b7b8c094af482d42dc87a6ff9941869177905"} Nov 03 22:07:55.040598 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:55.040540 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-588668ffc6-29dl9" podStartSLOduration=2.040483192 podStartE2EDuration="2.040483192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:55.039608279 +0000 UTC m=+68.119057321" watchObservedRunningTime="2026-11-03 22:07:55.040483192 +0000 UTC m=+68.119932227" Nov 03 22:07:55.934331 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:55.933816 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkldp" event={"ID":"03b9a551-4faa-46fb-8d85-9a39e39d9191","Type":"ContainerStarted","Data":"def15f2338aac7bd8d8b85dff8dfa9609ebcf8258960eb191cd5a75417a9a8d1"} Nov 03 22:07:55.934331 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:55.933879 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkldp" event={"ID":"03b9a551-4faa-46fb-8d85-9a39e39d9191","Type":"ContainerStarted","Data":"4d7c5aa90bfa4f876265b0d9ffbf4bb51a163d2aa1a70ab28b97767e6c3599a4"} Nov 03 22:07:56.051674 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:56.051429 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xkldp" podStartSLOduration=66.491084891 podStartE2EDuration="1m8.051409779s" totalImagesPullingTime="1.560324888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:56.048993224 +0000 UTC m=+69.128442261" watchObservedRunningTime="2026-11-03 22:07:56.051409779 +0000 UTC m=+69.130858815" Nov 03 22:07:56.941923 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:56.941888 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" event={"ID":"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d","Type":"ContainerStarted","Data":"9af38663d793aac2c2c4d1f67c90d5d08e503d12b20b00f66b26ece97a745a7f"} Nov 03 22:07:56.942420 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:56.942057 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" event={"ID":"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d","Type":"ContainerStarted","Data":"344d0e554770abde2a02e6d7741830fc8499ae16a4a3bc808935f5ec1452d8be"} Nov 03 22:07:57.166236 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:57.166180 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" podStartSLOduration=1.7680795900000001 podStartE2EDuration="6.166164207s" totalImagesPullingTime="4.398084617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:57.164784291 +0000 UTC m=+70.244233323" watchObservedRunningTime="2026-11-03 22:07:57.166164207 +0000 UTC m=+70.245613244" Nov 03 22:07:58.952081 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:58.952038 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9bqcf" event={"ID":"b595e184-27c8-4ffc-b50c-9764f013ef04","Type":"ContainerStarted","Data":"65cfae03204eb7b2ddee95a78f79d942d810080ca6fc2f93637ff705a2204061"} Nov 03 22:07:58.952556 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:58.952238 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:07:58.959435 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:58.958964 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rz85g" event={"ID":"9e6d3f43-3683-4c19-8b29-849e581ccd37","Type":"ContainerStarted","Data":"95c05e3c967d6dbea648291b79063ae6e3bf266789f4e0cd011937de89fb2325"} Nov 03 22:07:59.076339 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:59.076289 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9bqcf" podStartSLOduration=66.411264914 podStartE2EDuration="1m11.0762708s" totalImagesPullingTime="4.665005886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:59.07439671 +0000 UTC m=+72.153845772" watchObservedRunningTime="2026-11-03 22:07:59.0762708 +0000 UTC m=+72.155719837" Nov 03 22:07:59.189927 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:59.189862 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rz85g" podStartSLOduration=2.028050629 podStartE2EDuration="25.189843248s" totalImagesPullingTime="23.161792619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-03 22:07:59.189001839 +0000 UTC m=+72.268450874" watchObservedRunningTime="2026-11-03 22:07:59.189843248 +0000 UTC m=+72.269292286" Nov 03 22:07:59.932236 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:07:59.932208 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:00.353863 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:08:00.353839 2585 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-588668ffc6-29dl9"] Nov 03 22:08:00.529543 ip-10-0-134-57 kubenswrapper[2585]: I1103 22:08:00.529395 2585 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:08:00.529586 ip-10-0-134-57 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 22:08:00.541864 ip-10-0-134-57 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 22:08:00.542036 ip-10-0-134-57 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 22:08:00.542085 ip-10-0-134-57 systemd[1]: kubelet.service: Consumed 4.216s CPU time. Nov 03 22:08:00.555946 ip-10-0-134-57 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:08:00.617704 ip-10-0-134-57 kubenswrapper[6234]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:08:00.617704 ip-10-0-134-57 kubenswrapper[6234]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:08:00.617704 ip-10-0-134-57 kubenswrapper[6234]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:08:00.617704 ip-10-0-134-57 kubenswrapper[6234]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:08:00.619829 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619748 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:08:00.619829 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619828 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:08:00.619829 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619832 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619836 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619839 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619842 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619845 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619848 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619850 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619856 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619860 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619863 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619865 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619868 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619871 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619873 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619876 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619878 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619882 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619885 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619888 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:08:00.619939 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619891 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619893 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619896 6234 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619899 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619901 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619904 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619907 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619910 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619912 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619916 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619918 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619921 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619923 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619926 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619929 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619932 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619934 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619937 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619939 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619942 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:08:00.620450 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619945 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619948 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619950 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619953 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619956 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619959 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619962 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619964 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619967 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619970 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619973 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619976 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619978 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619981 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619984 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619987 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619990 6234 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619992 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619995 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.619998 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:08:00.620959 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620000 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620003 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620006 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620009 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620011 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620014 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620016 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620019 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620021 6234 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620024 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620028 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620030 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620035 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620039 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620042 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620045 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620048 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620050 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620053 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:08:00.621455 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620055 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620058 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620061 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620064 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620066 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620069 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620071 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620074 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620077 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620079 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620082 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620084 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620087 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620089 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620093 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620095 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620098 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620101 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620104 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:08:00.621979 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620107 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620111 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620115 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620118 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620121 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620124 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.620128 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621172 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621181 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621184 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621187 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621190 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621192 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621195 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621198 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621200 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621203 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621205 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621208 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621211 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:08:00.622641 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621218 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621221 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621223 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621226 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621228 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621231 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621234 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621237 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621239 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621242 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621245 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621248 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621251 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621254 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621258 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621262 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621265 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621267 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621270 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:08:00.623141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621273 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621275 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621278 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621281 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621283 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621286 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621289 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621291 6234 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621294 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621296 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621299 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621301 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621304 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621306 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621309 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621311 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621314 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621316 6234 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621319 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621321 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:08:00.623672 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621325 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621327 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621330 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621332 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621338 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621340 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621344 6234 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621347 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621350 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621352 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621355 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621357 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621360 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621362 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621365 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621367 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621370 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621373 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621375 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621378 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:08:00.624155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621380 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621383 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621385 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621388 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621390 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621393 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621395 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621398 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621401 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621403 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621406 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621410 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621414 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621417 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621420 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621422 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621426 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621429 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621431 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:08:00.624705 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621434 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621437 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621440 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621442 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621445 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621449 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621451 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621453 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621456 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621459 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621461 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621464 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621467 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621469 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621472 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621962 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621970 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621974 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621978 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621983 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:08:00.625161 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621986 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621989 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621992 6234 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621995 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.621998 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622001 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622003 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622006 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622009 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622012 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622016 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622020 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622023 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622026 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622028 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622031 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622034 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622044 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622047 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:08:00.625671 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622049 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622052 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622055 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622057 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622060 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622062 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622065 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622068 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622070 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622073 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622076 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622078 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622081 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622083 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622086 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622089 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622091 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622094 6234 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622096 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:08:00.626168 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622099 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622102 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622104 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622107 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622109 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622112 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622121 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622125 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622128 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622130 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622133 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622136 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622140 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622144 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622147 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622150 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622152 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622155 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622157 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622162 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:08:00.626852 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622165 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622171 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622175 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622180 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622184 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622188 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622192 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622201 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622206 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622211 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622215 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622219 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622222 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622226 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622230 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622235 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622239 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622243 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622250 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622254 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:08:00.627541 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622259 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622263 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622267 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622271 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622275 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622279 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622284 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622288 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622292 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622296 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622301 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622304 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622307 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622310 6234 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622312 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622315 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622318 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622320 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622323 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622326 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:08:00.628283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622328 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622331 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.622334 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622465 6234 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622485 6234 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622522 6234 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622530 6234 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622537 6234 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622542 6234 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622549 6234 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622557 6234 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622560 6234 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622564 6234 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622569 6234 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622572 6234 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622575 6234 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622578 6234 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622581 6234 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622585 6234 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622587 6234 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622590 6234 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622594 6234 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622597 6234 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622601 6234 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:08:00.628822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622604 6234 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622607 6234 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622613 6234 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622617 6234 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622620 6234 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622623 6234 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622626 6234 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622629 6234 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622632 6234 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622635 6234 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622638 6234 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622643 6234 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622646 6234 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622649 6234 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622652 6234 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622655 6234 flags.go:64] FLAG: --enable-server="true" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622659 6234 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622664 6234 flags.go:64] FLAG: --event-burst="100" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622668 6234 flags.go:64] FLAG: --event-qps="50" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622672 6234 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622675 6234 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622678 6234 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622683 6234 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622687 6234 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622690 6234 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:08:00.629396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622693 6234 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622696 6234 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622699 6234 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622702 6234 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622705 6234 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622708 6234 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622711 6234 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622714 6234 flags.go:64] FLAG: --feature-gates="" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622717 6234 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622722 6234 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622725 6234 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622728 6234 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622731 6234 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622734 6234 flags.go:64] FLAG: --help="false" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622737 6234 flags.go:64] FLAG: --hostname-override="ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622741 6234 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622744 6234 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622747 6234 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622750 6234 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622754 6234 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622756 6234 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622760 6234 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622762 6234 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:08:00.630102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622765 6234 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622768 6234 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622774 6234 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622777 6234 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622780 6234 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622783 6234 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622786 6234 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622789 6234 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622793 6234 flags.go:64] FLAG: --lock-file="" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622795 6234 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622798 6234 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622801 6234 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622807 6234 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622810 6234 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622813 6234 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622815 6234 flags.go:64] FLAG: --logging-format="text" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622818 6234 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622822 6234 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622826 6234 flags.go:64] FLAG: --manifest-url="" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622829 6234 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622833 6234 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622837 6234 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622840 6234 flags.go:64] FLAG: --max-pods="110" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622843 6234 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622846 6234 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:08:00.630743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622849 6234 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622852 6234 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622855 6234 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622858 6234 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622861 6234 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622869 6234 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622871 6234 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622874 6234 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622877 6234 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622882 6234 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622885 6234 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622888 6234 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622891 6234 flags.go:64] FLAG: --port="10250" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622895 6234 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622898 6234 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-005dede14741bddc3" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622902 6234 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622905 6234 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622908 6234 flags.go:64] FLAG: --register-node="true" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622910 6234 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622914 6234 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622917 6234 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622920 6234 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622923 6234 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622927 6234 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622930 6234 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622934 6234 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:08:00.631357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622937 6234 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622940 6234 flags.go:64] FLAG: --runonce="false" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622943 6234 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622946 6234 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622949 6234 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622952 6234 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622955 6234 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622958 6234 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622961 6234 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622964 6234 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622967 6234 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622970 6234 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622973 6234 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622976 6234 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622979 6234 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622984 6234 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622987 6234 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622992 6234 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622995 6234 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.622998 6234 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623002 6234 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623005 6234 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623014 6234 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623017 6234 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623020 6234 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623023 6234 flags.go:64] FLAG: --v="2" Nov 03 22:08:00.632069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623028 6234 flags.go:64] FLAG: --version="false" Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623032 6234 flags.go:64] FLAG: --vmodule="" Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623036 6234 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623039 6234 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623176 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623181 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623184 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623188 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623191 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623193 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623196 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623199 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623202 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623204 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623207 6234 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623210 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623213 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623215 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623218 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623221 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:08:00.632786 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623224 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623228 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623230 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623233 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623236 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623238 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623241 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623243 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623246 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623249 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623252 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623255 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623257 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623260 6234 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623262 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623265 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623268 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623272 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623274 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623278 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:08:00.633420 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623282 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623285 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623287 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623290 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623292 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623295 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623298 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623300 6234 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623302 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623305 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623308 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623311 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623314 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623317 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623320 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623323 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623325 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623328 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623330 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623333 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:08:00.634058 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623335 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623338 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623341 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623344 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623346 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623349 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623351 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623354 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623356 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623361 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623365 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623368 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623371 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623374 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623376 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623379 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623381 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623384 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623386 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623389 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:08:00.634877 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623391 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623394 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623396 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623399 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623401 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623405 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623408 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623411 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623413 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623416 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623419 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623421 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623424 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623426 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623429 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623432 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623434 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623437 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623440 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623443 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:08:00.635403 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623445 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623449 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623452 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623454 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623457 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623459 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623462 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623464 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623467 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623470 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.623510 6234 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623561 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623568 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623573 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623577 6234 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623583 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:08:00.635902 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623587 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623591 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623598 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623602 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623607 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623611 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623615 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623620 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623624 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623628 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623632 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623637 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623642 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623647 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623651 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623655 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623669 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623676 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623680 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:08:00.636328 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623683 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623688 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623692 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623695 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623698 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623701 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623703 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623706 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623709 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623712 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623714 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623717 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623719 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623722 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623725 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623729 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623731 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623734 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623736 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:08:00.636920 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623739 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623741 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623744 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623747 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623749 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623752 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623754 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623758 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623761 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623764 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623766 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623769 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623772 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623774 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623777 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623779 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623782 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623785 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623787 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:08:00.637438 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623790 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623792 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623795 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623797 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623802 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623804 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623807 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623810 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623813 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623817 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623820 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623823 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623825 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623828 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623830 6234 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623833 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623835 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623838 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623841 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623844 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:08:00.637994 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623847 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623850 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623853 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623856 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623858 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623861 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623863 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623866 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623869 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623871 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623874 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623876 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623879 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623882 6234 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623884 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623886 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623889 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623892 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623894 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623899 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:08:00.638471 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623902 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623914 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623918 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.623921 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.629028 6234 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.629046 6234 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.629065 6234 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.629070 6234 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629125 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629130 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629133 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629137 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629140 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629143 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629145 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629148 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629151 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629155 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629157 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629160 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:08:00.638975 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629163 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629166 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629168 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629171 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629178 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629180 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629183 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629186 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629189 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629191 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629194 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629198 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629201 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629204 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629207 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629210 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629212 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629215 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629218 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629220 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:08:00.639448 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629223 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629226 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629228 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629231 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629233 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629236 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629239 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629241 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629244 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629247 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629251 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629254 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629258 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629260 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629263 6234 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629266 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629269 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629272 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629274 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:08:00.639977 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629278 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629280 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629283 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629286 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629288 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629291 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629293 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629296 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629299 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629301 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629304 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629306 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629309 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629311 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629314 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629316 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629319 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629322 6234 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629324 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629327 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:08:00.640432 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629329 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629332 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629334 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629337 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629339 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629342 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629344 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629347 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629350 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629353 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629355 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629358 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629361 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629363 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629366 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629369 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629371 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629374 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629376 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:08:00.640938 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629379 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629382 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629384 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629387 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629389 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629392 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629395 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629398 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629401 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629404 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629406 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629409 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629412 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629414 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629417 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629419 6234 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:08:00.641535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.629447 6234 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629639 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629646 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629649 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629652 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629656 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629660 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629663 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629665 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629668 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629671 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629673 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629676 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629679 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629681 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629684 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629686 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629689 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629692 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:08:00.642081 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629694 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629697 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629699 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629702 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629706 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629709 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629712 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629716 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629719 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629722 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629724 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629727 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629729 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629732 6234 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629735 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629737 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629757 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629761 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629765 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629768 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:08:00.642742 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629771 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629774 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629777 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629779 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629782 6234 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629784 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629787 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629789 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629792 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629794 6234 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629797 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629799 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629802 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629805 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629807 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629810 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629812 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629815 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629817 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629820 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:08:00.643204 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629822 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629825 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629828 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629830 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629833 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629835 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629838 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629840 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629843 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629846 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629848 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629853 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629856 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629860 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629862 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629865 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629868 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629871 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629873 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629875 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:08:00.643756 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629878 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629881 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629883 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629886 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629890 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629894 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629897 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629900 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629902 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629905 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629907 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629910 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629912 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629915 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629917 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629920 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629922 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629925 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629927 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:08:00.644224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629930 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629933 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629935 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629938 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629942 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629944 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629947 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629951 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.629953 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.629978 6234 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630010 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630015 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630018 6234 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630021 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630024 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630026 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:08:00.644784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630029 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630032 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630034 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630037 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630039 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630042 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630044 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630047 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630050 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630052 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630055 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630057 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630060 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630063 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630065 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630068 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630070 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630073 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630076 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630078 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:08:00.645357 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630082 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630085 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630088 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630090 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630094 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630096 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630102 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630105 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630108 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630110 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630119 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630130 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630133 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630137 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630140 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630143 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630146 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630148 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630151 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:08:00.646222 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630154 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630156 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630159 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630162 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630164 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630167 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630170 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630172 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630175 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630177 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630180 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630182 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630185 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630189 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630192 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630194 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630197 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630200 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630203 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:08:00.646858 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630207 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630210 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630213 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630215 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630217 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630220 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630223 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630225 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630228 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630230 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630233 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630236 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630239 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630242 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630244 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630247 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630249 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630252 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630255 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:08:00.647447 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630258 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630262 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630265 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630268 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630271 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630274 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630276 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630280 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630283 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630285 6234 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630288 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630290 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630293 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630299 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630302 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630305 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630308 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630312 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630316 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630320 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:08:00.647978 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630324 6234 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630329 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:08:00.630333 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.630404 6234 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.631362 6234 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.631423 6234 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.631622 6234 server.go:1014] "Starting client certificate rotation" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.631774 6234 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.631827 6234 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:01:48 +0000 UTC" deadline="2028-07-22 18:40:56.483068314 +0000 UTC" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.631864 6234 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15044h32m55.851207644s" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.632161 6234 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.633222 6234 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.634430 6234 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.636392 6234 log.go:25] "Validated CRI v1 image API" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.637175 6234 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:08:00.648458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.639357 6234 fs.go:135] Filesystem UUIDs: map[0a876746-fc48-41f1-a8eb-5503b5d72f4c:/dev/nvme0n1p3 52e0cd8e-3ee6-403f-9500-c211b0842cda:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 22:08:00.648922 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.639377 6234 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08ce5f5336a6ce6e86809d215d3b7b8c094af482d42dc87a6ff9941869177905/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08ce5f5336a6ce6e86809d215d3b7b8c094af482d42dc87a6ff9941869177905/userdata/shm major:0 minor:419 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d0c414cba1b5256629dc7cd6b00f18c908e6f4e6d3ffedd9791bcf723e1df8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d0c414cba1b5256629dc7cd6b00f18c908e6f4e6d3ffedd9791bcf723e1df8d/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4422470e83279867e1ebbb50306c4393786557de2d6685d16207563890000f1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4422470e83279867e1ebbb50306c4393786557de2d6685d16207563890000f1b/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/450d028018fda0a97df38ed64931cf3a2bf63132fb0809cc40dcf1db702f8278/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/450d028018fda0a97df38ed64931cf3a2bf63132fb0809cc40dcf1db702f8278/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4607c7bf1180bce2fff71e4197866c36518890c83577c0021a5e656a19b850be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4607c7bf1180bce2fff71e4197866c36518890c83577c0021a5e656a19b850be/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48902d27bbc64df82c086b84be533621916319b412b2f122b5ae537d542f60af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48902d27bbc64df82c086b84be533621916319b412b2f122b5ae537d542f60af/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a118167180796978d0e8297a20c2a4554609e8740f3e3f4f2ebefbe142620c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a118167180796978d0e8297a20c2a4554609e8740f3e3f4f2ebefbe142620c9/userdata/shm major:0 minor:416 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a9054cd3149355ebd80f651406354a3552dea2bbfbbcd579869bf7e3c819c08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a9054cd3149355ebd80f651406354a3552dea2bbfbbcd579869bf7e3c819c08/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/517213ad3a35d48e0b4d1cbb0f2ba917e4d4d408dfc37bce2a3b9f9409a7d143/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/517213ad3a35d48e0b4d1cbb0f2ba917e4d4d408dfc37bce2a3b9f9409a7d143/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/613bda4db336d7790de31f19360018b21b4b764209444ee062d2f72aed278abc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/613bda4db336d7790de31f19360018b21b4b764209444ee062d2f72aed278abc/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/648fef1e7f821744d10df75f0b58cbcf3c3186a55452ba7ed725afd35a611762/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/648fef1e7f821744d10df75f0b58cbcf3c3186a55452ba7ed725afd35a611762/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6820ffc16879a9154d682d03811d75c42159650624ba2f39cffe69f6302210c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6820ffc16879a9154d682d03811d75c42159650624ba2f39cffe69f6302210c6/userdata/shm major:0 minor:439 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bc001e10574ea0052e18de1dcac1a0610084d34cee3c7266a718828f1b0fcda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bc001e10574ea0052e18de1dcac1a0610084d34cee3c7266a718828f1b0fcda/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6dda3470346bc8c16e5fdabb3a57d903822ac594ecc56d23e314e205703fa3b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6dda3470346bc8c16e5fdabb3a57d903822ac594ecc56d23e314e205703fa3b8/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ebc4734e6a8ed36070db7a8972f7749e3c517d9c80df80d2e7f82c1585dfc4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ebc4734e6a8ed36070db7a8972f7749e3c517d9c80df80d2e7f82c1585dfc4d/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/783772461ca1e801e5bb09b48acf897b15f24cc68d727f4339a3c24255a50f44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/783772461ca1e801e5bb09b48acf897b15f24cc68d727f4339a3c24255a50f44/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/824eadf4320ba290af7854d552fe68b2a4ac02727ca5e7fd8d16037d3e54c53f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/824eadf4320ba290af7854d552fe68b2a4ac02727ca5e7fd8d16037d3e54c53f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89c757a5702c30100e8ae8993aa9377cf1ddcbfda0678b6905d6e97e94a07355/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89c757a5702c30100e8ae8993aa9377cf1ddcbfda0678b6905d6e97e94a07355/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c6561d8dab64ac4f2a448bbb40c9cecd1c3cf2696d7af9e311496d39570f435/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c6561d8dab64ac4f2a448bbb40c9cecd1c3cf2696d7af9e311496d39570f435/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e0e7cbae6e8494573445cbdb079506a5bef9b6162dd8114a8e70fd96f9d482a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e0e7cbae6e8494573445cbdb079506a5bef9b6162dd8114a8e70fd96f9d482a/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ea7d018178e39398cebb348d374c83658c88d49ed28fb15c7bde2480c853ab4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ea7d018178e39398cebb348d374c83658c88d49ed28fb15c7bde2480c853ab4/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1ffd7459a3c94e519a3c1e82dc965b8c757c2ecd19c1956aad2a0bcf579e376/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1ffd7459a3c94e519a3c1e82dc965b8c757c2ecd19c1956aad2a0bcf579e376/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3230941b68331a7751394a36ddecba182890207f963e234e8ac5ca1399c038d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3230941b68331a7751394a36ddecba182890207f963e234e8ac5ca1399c038d/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2e3d6ffc8dedbdf6d54389ed6c94a671c7f96b9f3194c240958151422e18d54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2e3d6ffc8dedbdf6d54389ed6c94a671c7f96b9f3194c240958151422e18d54/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b584ea2c20c71c423c63f7de2148c394953c3292384e34d3caa883d5cab9013c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b584ea2c20c71c423c63f7de2148c394953c3292384e34d3caa883d5cab9013c/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f39aa78f21f42f418fb637667e7e4d006f900710474a682e9d6c43753bb1c552/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f39aa78f21f42f418fb637667e7e4d006f900710474a682e9d6c43753bb1c552/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02608c76-a776-4f06-94d3-eb3527a9dcdf/volumes/kubernetes.io~projected/kube-api-access-fg2lc:{mountpoint:/var/lib/kubelet/pods/02608c76-a776-4f06-94d3-eb3527a9dcdf/volumes/kubernetes.io~projected/kube-api-access-fg2lc major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02608c76-a776-4f06-94d3-eb3527a9dcdf/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/02608c76-a776-4f06-94d3-eb3527a9dcdf/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02608c76-a776-4f06-94d3-eb3527a9dcdf/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/02608c76-a776-4f06-94d3-eb3527a9dcdf/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b9a551-4faa-46fb-8d85-9a39e39d9191/volumes/kubernetes.io~projected/kube-api-access-qxdcf:{mountpoint:/var/lib/kubelet/pods/03b9a551-4faa-46fb-8d85-9a39e39d9191/volumes/kubernetes.io~projected/kube-api-access-qxdcf major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b9a551-4faa-46fb-8d85-9a39e39d9191/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/03b9a551-4faa-46fb-8d85-9a39e39d9191/volumes/kubernetes.io~secret/metrics-certs major:0 minor:406 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13fdc02e-6d8f-4e82-a8a4-9f295b64285d/volumes/kubernetes.io~projected/kube-api-access-tbclk:{mountpoint:/var/lib/kubelet/pods/13fdc02e-6d8f-4e82-a8a4-9f295b64285d/volumes/kubernetes.io~projected/kube-api-access-tbclk major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c056927-9f67-4a21-849b-dd71c606f245/volumes/kubernetes.io~projected/kube-api-access-xzspk:{mountpoint:/var/lib/kubelet/pods/1c056927-9f67-4a21-849b-dd71c606f245/volumes/kubernetes.io~projected/kube-api-access-xzspk major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c056927-9f67-4a21-849b-dd71c606f245/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1c056927-9f67-4a21-849b-dd71c606f245/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23923c80-00d0-4e1b-adbc-89cc9a992709/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/23923c80-00d0-4e1b-adbc-89cc9a992709/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433b8aa2-23d3-432b-abf6-15b7b334b287/volumes/kubernetes.io~projected/kube-api-access-nnp77:{mountpoint:/var/lib/kubelet/pods/433b8aa2-23d3-432b-abf6-15b7b334b287/volumes/kubernetes.io~projected/kube-api-access-nnp77 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433b8aa2-23d3-432b-abf6-15b7b334b287/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/433b8aa2-23d3-432b-abf6-15b7b334b287/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433b8aa2-23d3-432b-abf6-15b7b334b287/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/433b8aa2-23d3-432b-abf6-15b7b334b287/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~projected/kube-api-access-npwx8:{mountpoint:/var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~projected/kube-api-access-npwx8 major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b28053b-cac3-4543-a09b-25915e6b3f0c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5b28053b-cac3-4543-a09b-25915e6b3f0c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c1391f4-b5e9-4e91-88b3-208c56c2a153/volumes/kubernetes.io~projected/kube-api-access-9vp9t:{mountpoint:/var/lib/kubelet/pods/5c1391f4-b5e9-4e91-88b3-208c56c2a153/volumes/kubernetes.io~projected/kube-api-access-9vp9t major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~projected/kube-api-access-scj68:{mountpoint:/var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~projected/kube-api-access-scj68 major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48/volumes/kubernetes.io~projected/kube-api-access-s9qj7:{mountpoint:/var/lib/kubelet/pods/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48/volumes/kubernetes.io~projected/kube-api-access-s9qj7 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c201439-de30-4e86-88ca-c0826c25fe86/volumes/kubernetes.io~projected/kube-api-access-jk9dv:{mountpoint:/var/lib/kubelet/pods/6c201439-de30-4e86-88ca-c0826c25fe86/volumes/kubernetes.io~projected/kube-api-access-jk9dv major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/790226ec-d17d-47e7-b32f-da97c9e983c2/volumes/kubernetes.io~projected/kube-api-access-bpsm4:{mountpoint:/var/lib/kubelet/pods/790226ec-d17d-47e7-b32f-da97c9e983c2/volumes/kubernetes.io~projected/kube-api-access-bpsm4 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/790226ec-d17d-47e7-b32f-da97c9e983c2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/790226ec-d17d-47e7-b32f-da97c9e983c2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87e0c26d-7122-4e83-b875-35a4302cd6e1/volumes/kubernetes.io~projected/kube-api-access-9x8hb:{mountpoint:/var/lib/kubelet/pods/87e0c26d-7122-4e83-b875-35a4302cd6e1/volumes/kubernetes.io~projected/kube-api-access-9x8hb major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e6d3f43-3683-4c19-8b29-849e581ccd37/volumes/kubernetes.io~projected/kube-api-access-kz5kc:{mountpoint:/var/lib/kubelet/pods/9e6d3f43-3683-4c19-8b29-849e581ccd37/volumes/kubernetes.io~projected/kube-api-access-kz5kc major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e6d3f43-3683-4c19-8b29-849e581ccd37/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9e6d3f43-3683-4c19-8b29-849e581ccd37/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0f26ebc-df35-404e-9466-9f5d73478ef7/volumes/kubernetes.io~projected/kube-api-access-2cpph:{mountpoint:/var/lib/kubelet/pods/a0f26ebc-df35-404e-9466-9f5d73478ef7/volumes/kubernetes.io~projected/kube-api-access-2cpph major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0f26ebc-df35-404e-9466-9f5d73478ef7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a0f26ebc-df35-404e-9466-9f5d73478ef7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0f26ebc-df35-404e-9466-9f5d73478ef7/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/a0f26ebc-df35-404e-9466-9f5d73478ef7/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a344707b-bc31-4587-ad1b-5da15d0a4dc9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a344707b-bc31-4587-ad1b-5da15d0a4dc9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a344707b-bc31-4587-ad1b-5da15d0a4dc9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a344707b-bc31-4587-ad1b-5da15d0a4dc9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a344707b-bc31-4587-ad1b-5da15d0a4dc9/volumes/kubernetes.io~projected/kube-api-access-wxdqr:{mountpoint:/var/lib/kubelet/pods/a344707b-bc31-4587-ad1b-5da15d0a4dc9/volumes/kubernetes.io~projected/kube-api-access-wxdqr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b595e184-27c8-4ffc-b50c-9764f013ef04/volumes/kubernetes.io~projected/kube-api-access-w4gz5:{mountpoint:/var/lib/kubelet/pods/b595e184-27c8-4ffc-b50c-9764f013ef04/volumes/kubernetes.io~projected/kube-api-access-w4gz5 major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/beccabec-c5c9-4f17-a8ba-c23e1254c391/volumes/kubernetes.io~projected/kube-api-access-9l9x4:{mountpoint:/var/lib/kubelet/pods/beccabec-c5c9-4f17-a8ba-c23e1254c391/volumes/kubernetes.io~projected/kube-api-access-9l9x4 major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/beccabec-c5c9-4f17-a8ba-c23e1254c391/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/beccabec-c5c9-4f17-a8ba-c23e1254c391/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/beccabec-c5c9-4f17-a8ba-c23e1254c391/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/beccabec-c5c9-4f17-a8ba-c23e1254c391/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9d22708-0a5b-41de-bd34-78e11212e865/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/c9d22708-0a5b-41de-bd34-78e11212e865/volumes/kubernetes.io~secret/tls-certificates major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~projected/kube-api-access-9nrm4:{mountpoint:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~projected/kube-api-access-9nrm4 major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e40ade3f-f022-48c6-b9ed-a219100b4ff2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e40ade3f-f022-48c6-b9ed-a219100b4ff2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e40ade3f-f022-48c6-b9ed-a219100b4ff2/volumes/kubernetes.io~projected/kube-api-access-ggwr4:{mountpoint:/var/lib/kubelet/pods/e40ade3f-f022-48c6-b9ed-a219100b4ff2/volumes/kubernetes.io~projected/kube-api-access-ggwr4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e40ade3f-f022-48c6-b9ed-a219100b4ff2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e40ade3f-f022-48c6-b9ed-a219100b4ff2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec96604b-256b-404b-95d1-5fe5cb00a147/volumes/kubernetes.io~projected/kube-api-access-nclln:{mountpoint:/var/lib/kubelet/pods/ec96604b-256b-404b-95d1-5fe5cb00a147/volumes/kubernetes.io~projected/kube-api-access-nclln major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed75a83a-9f07-4a2f-be7f-1a0b249728c0/volumes/kubernetes.io~projected/kube-api-access-64bmn:{mountpoint:/var/lib/kubelet/pods/ed75a83a-9f07-4a2f-be7f-1a0b249728c0/volumes/kubernetes.io~projected/kube-api-access-64bmn major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed75a83a-9f07-4a2f-be7f-1a0b249728c0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ed75a83a-9f07-4a2f-be7f-1a0b249728c0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed75a83a-9f07-4a2f-be7f-1a0b249728c0/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ed75a83a-9f07-4a2f-be7f-1a0b249728c0/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd82c8e7-5bc3-4cb7-be68-a519c025f61f/volumes/kubernetes.io~projected/kube-api-access-qlgzg:{mountpoint:/var/lib/kubelet/pods/fd82c8e7-5bc3-4cb7-be68-a519c025f61f/volumes/kubernetes.io~projected/kube-api-access-qlgzg major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ab35fa0f3dd88165a11f5b148d99a94b1b791d33a503d8b3e0a8edb8111ba758/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/27c0c55a48e2aa88323d22cc877ba2ed8d656c7f62426fd5fa5fe8dc88fbb9fe/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/32fa3e5828fdf131e812dfe89208de1e31948839dd0428973b4e40c6710cd3cf/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/3d26b74b8567ba7ba39ca582c78f26b4b41ec461f3318eb1c6c8a69b76e2ee5b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8ac893f289e83434b63d455dfd6061d898efc81f562e17f51a88e566bdfaccd7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c0e3bc95d15727555f2d11f5ea42c12811a73e032ea897246066bb3a92c3b6aa/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3fd6b08ebce0b8ad7d46a3f87f5a99651c58aac488da9b8e132bac454d38fbaf/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/50e0b04b290d52608d789caf753c268da13fa389174b316d8730533fd65b5be1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/02f184c20b96d80586e02f0e79ab676aa0dd8760d73262e1fdce28ba64e00ecc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0dca2140187cbcae3ddf22220936e38747b7435a866efcb5fe62a833268017cb/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b3f0e4b08671437d919c5db392c955025fda59c2caa7c7399ac82a455ebe963b/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f9caed3414be0c5f53a1327463f9d5ad8ed993a98d93d1425d9cab13067cea39/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/21cd1bfad217ae512dfba43f08864938257f7bd688fb2bd6581eeb6752cfe42e/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/45d01c3a950e3ebc87849451833750220f655b66273664bc4dee81996ce727d0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/79b47ad8c9718d2e3cce06acafcede18b39e23430e10a0199aa217a1f4b1ca07/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/93eeb707e3717e1cb8b987f95e3fffe2ba3ec1a5476dcaaf3f50badbbda15d54/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/82a1d0e9f9b97a4c9a78bf76724a155b5a7f8d1f28147b034978da2a10fcc729/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f1db7ce151d9357c65bf0de77977e2fbf9ff75b3ff0d19c0f8dadcaee339f16e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/079f30c143874c893c9975b3ecaaa1b5dcdaa830a2218a46ce45eb08e3a7351c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0392b9c70592432c01eba5002af99fe6e2d879146b7eedfd119e879e27b9c6e2/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/18f02c3f2845c58e6a105850c434231b60fa33c668ca4a9bb0aca374adb8402c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/5b5ce15f02fbe7c16c14a171653627559df0e4f5b447016c36bacb6b5ae205e6/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/3c56a1802f08d980acd3abe9e9ac5840699d9ce38af5213fad164481033cc9d2/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/08364b533bef15ffe275d172c7697ad367f5d273ec703901ab3b8df05a3cebbc/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/6dd6686f2097f9e3ab53a28ef445fbaaadfb9a521e456b11b84c140ecc8a8f8b/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/9fc460a2353bceb8d85f03650f697558096a6f395080c9f44488a3008b6667a5/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/39dea21e3ba7e0c9212d104b1128cbb3bcb3e2ab2c778e518cdec60282305b4b/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/407a40dadfb7a20d769db6fb4d0e28b4ffd78dc44884e74cf341fffcf74f35f5/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/ccbaef00918528b9475f8c764c9f2ddc9b2025c817618f776681ce3b38f248bc/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/fbf0a1800fa4ca644854fe518a494f3433f1039651998c8b7e0e0f27bf8d4764/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/1debabe119b5ff6ad1c491be655b5842a0777d58d44c57ba0f90ba4bf5f24de6/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/6a9b9a6a95e4ed6cc8ff3c4c2eabd8788c8e870fe140973b9c248cf4b0fade02/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/00ccfdd2ed5b289119f81e278267dffa1e4b392f8e62c42d04f2ea747fe7152a/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/68001e2195faaab18622a61ffdaee0088eacb6d30921e0ab7676799f4439d41c/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/8ca29b33741f54cd00472cb857546bd85feb14598f3183c8b55fed1c9b7da141/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/c063138e715af5d5aa34b1627d20d096c013e66cf18feb6fab75e2d4855c94d1/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/0753a6c2c8d8b15f6f129d3d467d89645dd6a4085310b8bcd917981bb72af327/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/a2ce930e7402d5de72140f67288e8b9c5c4384f7c1e7cc23071bfaa6e96d71f4/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/e47ea167073320e824a2325135e77603cd825db213fc1740e0419b2e7886b330/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/744e76c43ca7ae5fff52e313ad87697813a4058f86e1c567e7c3f5a86d6fd5bf/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/58c6469d56fa6236bf1e66ae4030d95cc855d6519fb5286c6a2eae6492246552/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/0cc042024ae0dd9c1fc22095420abc6d5ad0bfaeefea591f28202ec4f2ce3b5c/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/288f920778814273de7e8675f9fc798f6a21cf4a268e0eed9201f83cbf458338/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/ba92999868ed471ab5c4f43f94f9f04d571111ef24257331b7907561a84db310/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/73fb9828e920a1616e2ebb059cdbc404bef0045538d681e2c9478a4550131817/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/65f254582c047b6fcced05499126751e612ef32cb925a3f15a939b58fc6c4dbf/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/c81fe4b3cab0e329e24fc0dc5c9b46a54b531db1cce4f5de5748a11a06f928f2/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/c023b209ebffcaf2cdbf26e303075c2679d72224fd8a0ee03aca1d64ed68df3b/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/d4ac8b73742429e61dd31485197bc851bfad3219ced8ede8fac591c0f7d78877/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/0c68fdbb811cd232448af39c5ff7224e7a6cbe6ec0eb3b11a5959fcfd9be468d/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/388eb96518b62b66516c545b0d923b5810602fb37a72620fea15c0e1ad5d43bb/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/24f8f8140968a8b282a9f13e4e8f4e86b353f50fde3f3a76ba511a89137637a6/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/c6591f821ae5ff7cc7195cc8e058357b1c5ffebb427c66f43b1371d80d4882a9/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/d5a68321326734c6e563498d0528021dc077bc5fe8c481cd0f8d9ff3d20f4409/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/24632af3debbe33edf043d0c3f62dff96936e329275792c589781dcfcec91b55/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/fc21795ea085255c9452de1a595694b48da8c5b13e067c1aa56442ebe2b2b2c6/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/4e7613a894822aa7451105e276ba52bbeb082bd07f53ab08a24f97dfe52e3394/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/a4d8370cbd5907058ca710e91eaa6746a6a8004d7c64ad561adf4384b5912e74/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/c210c9836b51ca4ff837a9cd03b309295d71070c9b41fb2160e491e282503a07/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/ac18de2b312114e1f1b6cbc6c65c6caeed5faadf5d1b61d5507eb5eeb26f4897/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/af03c0dfa4fc1bdfa80d8a9007cf892fc68b0019ecf33464e074e82de2eca73e/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/2b6e7f169eaa257cef8f8103c57fbdce734958de0b39a07749b7479fc99ea9c6/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/6c9376ce8beb34279eab89c4f5d6ba66b6824b63359ad85ecfd2415988da891e/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/2a922ffafb80f23a9461d8e90bf69e20caa9b4e97293416a5aa57f37b0845f29/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/1afa9a2a1a125a8cd91629ae3b342e8e3adaceedbbaca78e624e08fa67e2dfa2/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/44dc45fdbe9288531ce48ed0aac5d1dc4a006519387d694553071c721258d7e1/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/284b6625a2bbb5becb7ed71469be58e40570352520c3a6027d1037965b2ba678/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/85e6b64fd788138d433e3ea46777378dadcc8321af44026d4b71894bea2064f5/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/b388dd6f7eafeaafe94afae66a515c900f72e4589c6e55ca312e249181d81acb/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/fa1750c0909974c9175af13b42349b98ead2b494e21d6e7a62e74ae2e36e02f9/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/ba44f3421f5d83df8d5af34b113c281c4e6cb72ad399e9ae28c0e41642e0fc43/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c724301e594545d452a439144620b8807ace8ddec6321b08ad804bbf9f7f9cf6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2e1a7c8bdbbc22b3d9b54841bd9f8599120635209bd9b9693de7959e03bf8eec/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8de9ecd7de06893452bc9dd9d77073e81d82a1849d27ef8b8e930cf32e9cb9a5/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f40a59168743545e051c0ec0ec5df7a1064fad202f8b93eb8a9960726d89c785/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/eeb957bb3ec448abeda6d119e631675769a6815c390ebaec29859a405fba4dfd/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/5132a2b275cfb4e5ab2251afa149692c144579bb482b9b10610ca008e481fc46/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4c0d827a7b4a6a6902a8d425db3a5e6bd1a75dbe342a23a6cdbd6e088fb46d7e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8df724752242fb51fadcfe8385898b3fb95fc354f487764c46db7f29182035f4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bce45be7096375522c0e3dfca4ea9a07ec525a9a315fb9e3c6907d2219d28984/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.647859 6234 manager.go:217] Machine: {Timestamp:2026-11-03 22:08:00.647383693 +0000 UTC m=+0.059569701 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101886 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa114374c4bc49556fa5e80d5a785 SystemUUID:ec2aa114-374c-4bc4-9556-fa5e80d5a785 BootID:27453300-eb66-4d36-a070-ab89f7d83c8d Filesystems:[{Device:/var/lib/kubelet/pods/9e6d3f43-3683-4c19-8b29-849e581ccd37/volumes/kubernetes.io~projected/kube-api-access-kz5kc DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08ce5f5336a6ce6e86809d215d3b7b8c094af482d42dc87a6ff9941869177905/userdata/shm DeviceMajor:0 DeviceMinor:419 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d0c414cba1b5256629dc7cd6b00f18c908e6f4e6d3ffedd9791bcf723e1df8d/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a344707b-bc31-4587-ad1b-5da15d0a4dc9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~projected/kube-api-access-scj68 DeviceMajor:0 DeviceMinor:372 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd82c8e7-5bc3-4cb7-be68-a519c025f61f/volumes/kubernetes.io~projected/kube-api-access-qlgzg DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e0e7cbae6e8494573445cbdb079506a5bef9b6162dd8114a8e70fd96f9d482a/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed75a83a-9f07-4a2f-be7f-1a0b249728c0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:281 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b595e184-27c8-4ffc-b50c-9764f013ef04/volumes/kubernetes.io~projected/kube-api-access-w4gz5 DeviceMajor:0 DeviceMinor:418 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/02608c76-a776-4f06-94d3-eb3527a9dcdf/volumes/kubernetes.io~projected/kube-api-access-fg2lc DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c056927-9f67-4a21-849b-dd71c606f245/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f39aa78f21f42f418fb637667e7e4d006f900710474a682e9d6c43753bb1c552/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9d22708-0a5b-41de-bd34-78e11212e865/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02608c76-a776-4f06-94d3-eb3527a9dcdf/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/beccabec-c5c9-4f17-a8ba-c23e1254c391/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:429 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6820ffc16879a9154d682d03811d75c42159650624ba2f39cffe69f6302210c6/userdata/shm DeviceMajor:0 DeviceMinor:439 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b584ea2c20c71c423c63f7de2148c394953c3292384e34d3caa883d5cab9013c/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~projected/kube-api-access-npwx8 DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/433b8aa2-23d3-432b-abf6-15b7b334b287/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/433b8aa2-23d3-432b-abf6-15b7b334b287/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ebc4734e6a8ed36070db7a8972f7749e3c517d9c80df80d2e7f82c1585dfc4d/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3230941b68331a7751394a36ddecba182890207f963e234e8ac5ca1399c038d/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/02608c76-a776-4f06-94d3-eb3527a9dcdf/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:371 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e40ade3f-f022-48c6-b9ed-a219100b4ff2/volumes/kubernetes.io~projected/kube-api-access-ggwr4 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0f26ebc-df35-404e-9466-9f5d73478ef7/volumes/kubernetes.io~projected/kube-api-access-2cpph DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4422470e83279867e1ebbb50306c4393786557de2d6685d16207563890000f1b/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2e3d6ffc8dedbdf6d54389ed6c94a671c7f96b9f3194c240958151422e18d54/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c6561d8dab64ac4f2a448bbb40c9cecd1c3cf2696d7af9e311496d39570f435/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/450d028018fda0a97df38ed64931cf3a2bf63132fb0809cc40dcf1db702f8278/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e40ade3f-f022-48c6-b9ed-a219100b4ff2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9e6d3f43-3683-4c19-8b29-849e581ccd37/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0f26ebc-df35-404e-9466-9f5d73478ef7/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5c1391f4-b5e9-4e91-88b3-208c56c2a153/volumes/kubernetes.io~projected/kube-api-access-9vp9t DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/613bda4db336d7790de31f19360018b21b4b764209444ee062d2f72aed278abc/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:274 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a9054cd3149355ebd80f651406354a3552dea2bbfbbcd579869bf7e3c819c08/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48902d27bbc64df82c086b84be533621916319b412b2f122b5ae537d542f60af/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/13fdc02e-6d8f-4e82-a8a4-9f295b64285d/volumes/kubernetes.io~projected/kube-api-access-tbclk DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/824eadf4320ba290af7854d552fe68b2a4ac02727ca5e7fd8d16037d3e54c53f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5692356c-6609-46d4-9cd7-513a0d883779/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:362 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a344707b-bc31-4587-ad1b-5da15d0a4dc9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/648fef1e7f821744d10df75f0b58cbcf3c3186a55452ba7ed725afd35a611762/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed75a83a-9f07-4a2f-be7f-1a0b249728c0/volumes/kubernetes.io~projected/kube-api-access-64bmn DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:368 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ea7d018178e39398cebb348d374c83658c88d49ed28fb15c7bde2480c853ab4/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/beccabec-c5c9-4f17-a8ba-c23e1254c391/volumes/kubernetes.io~projected/kube-api-access-9l9x4 DeviceMajor:0 DeviceMinor:431 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ec96604b-256b-404b-95d1-5fe5cb00a147/volumes/kubernetes.io~projected/kube-api-access-nclln DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/790226ec-d17d-47e7-b32f-da97c9e983c2/volumes/kubernetes.io~projected/kube-api-access-bpsm4 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bc001e10574ea0052e18de1dcac1a0610084d34cee3c7266a718828f1b0fcda/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c056927-9f67-4a21-849b-dd71c606f245/volumes/kubernetes.io~projected/kube-api-access-xzspk DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/783772461ca1e801e5bb09b48acf897b15f24cc68d727f4339a3c24255a50f44/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48/volumes/kubernetes.io~projected/kube-api-access-s9qj7 DeviceMajor:0 DeviceMinor:275 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89c757a5702c30100e8ae8993aa9377cf1ddcbfda0678b6905d6e97e94a07355/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/790226ec-d17d-47e7-b32f-da97c9e983c2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4607c7bf1180bce2fff71e4197866c36518890c83577c0021a5e656a19b850be/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1ffd7459a3c94e519a3c1e82dc965b8c757c2ecd19c1956aad2a0bcf579e376/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/beccabec-c5c9-4f17-a8ba-c23e1254c391/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:430 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/517213ad3a35d48e0b4d1cbb0f2ba917e4d4d408dfc37bce2a3b9f9409a7d143/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87e0c26d-7122-4e83-b875-35a4302cd6e1/volumes/kubernetes.io~projected/kube-api-access-9x8hb DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23923c80-00d0-4e1b-adbc-89cc9a992709/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/433b8aa2-23d3-432b-abf6-15b7b334b287/volumes/kubernetes.io~projected/kube-api-access-nnp77 DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e40ade3f-f022-48c6-b9ed-a219100b4ff2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a118167180796978d0e8297a20c2a4554609e8740f3e3f4f2ebefbe142620c9/userdata/shm DeviceMajor:0 DeviceMinor:416 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a344707b-bc31-4587-ad1b-5da15d0a4dc9/volumes/kubernetes.io~projected/kube-api-access-wxdqr DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d962bd32-3fc5-4a46-ba44-0d08701a6cd0/volumes/kubernetes.io~projected/kube-api-access-9nrm4 DeviceMajor:0 DeviceMinor:346 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed75a83a-9f07-4a2f-be7f-1a0b249728c0/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/03b9a551-4faa-46fb-8d85-9a39e39d9191/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:406 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0f26ebc-df35-404e-9466-9f5d73478ef7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03b9a551-4faa-46fb-8d85-9a39e39d9191/volumes/kubernetes.io~projected/kube-api-access-qxdcf DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c201439-de30-4e86-88ca-c0826c25fe86/volumes/kubernetes.io~projected/kube-api-access-jk9dv DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6dda3470346bc8c16e5fdabb3a57d903822ac594ecc56d23e314e205703fa3b8/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:370 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b28053b-cac3-4543-a09b-25915e6b3f0c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:412 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:08ce5f5336a6ce6 MacAddress:ba:d6:26:d9:9d:76 Speed:10000 Mtu:8901} {Name:4422470e8327986 MacAddress:2e:db:57:7b:ff:40 Speed:10000 Mtu:8901} {Name:450d028018fda0a MacAddress:36:72:33:c8:7d:5a Speed:10000 Mtu:8901} {Name:4607c7bf1180bce MacAddress:92:35:dc:e0:ab:77 Speed:10000 Mtu:8901} {Name:4a1181671807969 MacAddress:fa:c1:81:f7:0c:d8 Speed:10000 Mtu:8901} {Name:4a9054cd3149355 MacAddress:d2:9b:07:18:2c:4f Speed:10000 Mtu:8901} {Name:6820ffc16879a91 MacAddress:0a:fc:ba:c5:b2:9a Speed:10000 Mtu:8901} {Name:6dda3470346bc8c MacAddress:52:a7:5b:7c:84:10 Speed:10000 Mtu:8901} {Name:6ebc4734e6a8ed3 MacAddress:e2:5d:de:74:87:21 Speed:10000 Mtu:8901} {Name:783772461ca1e80 MacAddress:b2:41:83:b7:a6:9d Speed:10000 Mtu:8901} {Name:9e0e7cbae6e8494 MacAddress:42:bb:ca:ef:1d:4d Speed:10000 Mtu:8901} {Name:9ea7d018178e393 MacAddress:5a:5d:b8:e9:32:e3 Speed:10000 Mtu:8901} {Name:a3230941b68331a MacAddress:1e:56:e3:6c:42:31 Speed:10000 Mtu:8901} {Name:b584ea2c20c71c4 MacAddress:56:7e:36:52:57:a2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:cc:fd:92:11 Speed:0 Mtu:9001} {Name:br-int MacAddress:22:73:e7:ab:e0:4d Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:cc:fd:92:11 Speed:0 Mtu:9001} {Name:f39aa78f21f42f4 MacAddress:22:5a:07:e6:e9:eb Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:da:e2:ac:88:d6:bc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:34:c9:8a:83:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.648278 6234 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.648322 6234 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.648569 6234 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.648590 6234 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-57.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.648707 6234 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.648715 6234 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.648725 6234 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.648745 6234 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.648816 6234 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.648952 6234 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.649049 6234 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.649080 6234 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.649092 6234 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.649101 6234 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.649109 6234 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:08:00.649232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.649117 6234 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:08:00.650136 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.649736 6234 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:08:00.650136 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.649746 6234 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:08:00.650204 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.650157 6234 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:08:00.652870 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.652854 6234 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:08:00.652870 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.652867 6234 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:08:00.652968 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.652950 6234 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 22:08:00.653129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653118 6234 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:08:00.653170 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653147 6234 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:08:00.653241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653232 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:08:00.653282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653246 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:08:00.653282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653253 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:08:00.653282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653258 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:08:00.653282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653265 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:08:00.653282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653271 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:08:00.653282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653276 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:08:00.653282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653282 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:08:00.653671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653288 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:08:00.653671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653296 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:08:00.653671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653304 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:08:00.653671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653313 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:08:00.653671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653337 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:08:00.653671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.653343 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:08:00.656601 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.656586 6234 server.go:1263] "Started kubelet" Nov 03 22:08:00.656722 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.656688 6234 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:08:00.656784 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.656687 6234 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:08:00.656784 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.656780 6234 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:08:00.657106 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.657087 6234 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:08:00.657170 ip-10-0-134-57 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:08:00.660845 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.659698 6234 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:08:00.662961 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.662748 6234 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:08:00.662961 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.662858 6234 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:08:00.664304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.663132 6234 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:01:50 +0000 UTC" deadline="2028-08-16 08:25:52.212900534 +0000 UTC" Nov 03 22:08:00.664304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.663177 6234 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15634h17m51.549728144s" Nov 03 22:08:00.664304 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:00.663242 6234 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:08:00.664304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.663274 6234 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:08:00.664304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.663281 6234 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:08:00.664304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.663541 6234 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:08:00.664648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.664322 6234 factory.go:55] Registering systemd factory Nov 03 22:08:00.664648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.664342 6234 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:08:00.664648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.664549 6234 factory.go:153] Registering CRI-O factory Nov 03 22:08:00.664648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.664561 6234 factory.go:223] Registration of the crio container factory successfully Nov 03 22:08:00.664648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.664626 6234 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:08:00.664648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.664647 6234 factory.go:103] Registering Raw factory Nov 03 22:08:00.664925 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.664664 6234 manager.go:1199] Started watching for new ooms in manager Nov 03 22:08:00.665018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.665002 6234 manager.go:319] Starting recovery of all containers Nov 03 22:08:00.666797 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666762 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" volumeName="kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config" seLinuxMountContext="" Nov 03 22:08:00.666881 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666803 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="790226ec-d17d-47e7-b32f-da97c9e983c2" volumeName="kubernetes.io/projected/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-api-access-bpsm4" seLinuxMountContext="" Nov 03 22:08:00.666881 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666820 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a344707b-bc31-4587-ad1b-5da15d0a4dc9" volumeName="kubernetes.io/projected/a344707b-bc31-4587-ad1b-5da15d0a4dc9-kube-api-access-wxdqr" seLinuxMountContext="" Nov 03 22:08:00.666881 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666833 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beccabec-c5c9-4f17-a8ba-c23e1254c391" volumeName="kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert" seLinuxMountContext="" Nov 03 22:08:00.666881 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666846 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962bd32-3fc5-4a46-ba44-0d08701a6cd0" volumeName="kubernetes.io/empty-dir/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-tmp" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666894 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed75a83a-9f07-4a2f-be7f-1a0b249728c0" volumeName="kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666930 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c1391f4-b5e9-4e91-88b3-208c56c2a153" volumeName="kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666946 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d71ea07-fa84-4b2a-b06d-f7b05d7a126d" volumeName="kubernetes.io/projected/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-kube-api-access-scj68" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666962 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d71ea07-fa84-4b2a-b06d-f7b05d7a126d" volumeName="kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-federate-client-tls" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666974 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" volumeName="kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.666988 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02608c76-a776-4f06-94d3-eb3527a9dcdf" volumeName="kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667001 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433b8aa2-23d3-432b-abf6-15b7b334b287" volumeName="kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-metrics-client-ca" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667013 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5692356c-6609-46d4-9cd7-513a0d883779" volumeName="kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667028 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0f26ebc-df35-404e-9466-9f5d73478ef7" volumeName="kubernetes.io/projected/a0f26ebc-df35-404e-9466-9f5d73478ef7-kube-api-access-2cpph" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667043 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5692356c-6609-46d4-9cd7-513a0d883779" volumeName="kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 22:08:00.667075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667071 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" volumeName="kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667088 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e6d3f43-3683-4c19-8b29-849e581ccd37" volumeName="kubernetes.io/projected/9e6d3f43-3683-4c19-8b29-849e581ccd37-kube-api-access-kz5kc" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667102 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beccabec-c5c9-4f17-a8ba-c23e1254c391" volumeName="kubernetes.io/empty-dir/beccabec-c5c9-4f17-a8ba-c23e1254c391-tmp" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667115 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e40ade3f-f022-48c6-b9ed-a219100b4ff2" volumeName="kubernetes.io/secret/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667127 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec96604b-256b-404b-95d1-5fe5cb00a147" volumeName="kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-cni-binary-copy" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667138 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec96604b-256b-404b-95d1-5fe5cb00a147" volumeName="kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-daemon-config" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667152 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed75a83a-9f07-4a2f-be7f-1a0b249728c0" volumeName="kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667165 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02608c76-a776-4f06-94d3-eb3527a9dcdf" volumeName="kubernetes.io/empty-dir/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-textfile" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667179 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fdc02e-6d8f-4e82-a8a4-9f295b64285d" volumeName="kubernetes.io/empty-dir/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-tmp-dir" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667208 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c056927-9f67-4a21-849b-dd71c606f245" volumeName="kubernetes.io/configmap/1c056927-9f67-4a21-849b-dd71c606f245-config-volume" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667221 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c056927-9f67-4a21-849b-dd71c606f245" volumeName="kubernetes.io/empty-dir/1c056927-9f67-4a21-849b-dd71c606f245-tmp-dir" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667235 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962bd32-3fc5-4a46-ba44-0d08701a6cd0" volumeName="kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667248 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fdc02e-6d8f-4e82-a8a4-9f295b64285d" volumeName="kubernetes.io/projected/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-kube-api-access-tbclk" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667267 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c056927-9f67-4a21-849b-dd71c606f245" volumeName="kubernetes.io/secret/1c056927-9f67-4a21-849b-dd71c606f245-metrics-tls" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667278 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b28053b-cac3-4543-a09b-25915e6b3f0c" volumeName="kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667290 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c1391f4-b5e9-4e91-88b3-208c56c2a153" volumeName="kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667303 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="790226ec-d17d-47e7-b32f-da97c9e983c2" volumeName="kubernetes.io/configmap/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667317 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0f26ebc-df35-404e-9466-9f5d73478ef7" volumeName="kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667329 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a344707b-bc31-4587-ad1b-5da15d0a4dc9" volumeName="kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-tmp" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667342 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c1391f4-b5e9-4e91-88b3-208c56c2a153" volumeName="kubernetes.io/projected/5c1391f4-b5e9-4e91-88b3-208c56c2a153-kube-api-access-9vp9t" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667353 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" volumeName="kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667364 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beccabec-c5c9-4f17-a8ba-c23e1254c391" volumeName="kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667375 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9d22708-0a5b-41de-bd34-78e11212e865" volumeName="kubernetes.io/secret/c9d22708-0a5b-41de-bd34-78e11212e865-tls-certificates" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667395 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962bd32-3fc5-4a46-ba44-0d08701a6cd0" volumeName="kubernetes.io/configmap/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-metrics-client-ca" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667402 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed75a83a-9f07-4a2f-be7f-1a0b249728c0" volumeName="kubernetes.io/configmap/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-metrics-client-ca" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667410 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23923c80-00d0-4e1b-adbc-89cc9a992709" volumeName="kubernetes.io/configmap/23923c80-00d0-4e1b-adbc-89cc9a992709-konnectivity-ca" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667417 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433b8aa2-23d3-432b-abf6-15b7b334b287" volumeName="kubernetes.io/projected/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-api-access-nnp77" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667425 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5692356c-6609-46d4-9cd7-513a0d883779" volumeName="kubernetes.io/empty-dir/5692356c-6609-46d4-9cd7-513a0d883779-audit-log" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667435 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d71ea07-fa84-4b2a-b06d-f7b05d7a126d" volumeName="kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-metrics-client-ca" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667447 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" volumeName="kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667458 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0f26ebc-df35-404e-9466-9f5d73478ef7" volumeName="kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-tls" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667471 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beccabec-c5c9-4f17-a8ba-c23e1254c391" volumeName="kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667484 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962bd32-3fc5-4a46-ba44-0d08701a6cd0" volumeName="kubernetes.io/projected/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-kube-api-access-9nrm4" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667518 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02608c76-a776-4f06-94d3-eb3527a9dcdf" volumeName="kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667532 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962bd32-3fc5-4a46-ba44-0d08701a6cd0" volumeName="kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667542 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962bd32-3fc5-4a46-ba44-0d08701a6cd0" volumeName="kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667561 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d71ea07-fa84-4b2a-b06d-f7b05d7a126d" volumeName="kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667569 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d71ea07-fa84-4b2a-b06d-f7b05d7a126d" volumeName="kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-client-tls" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667577 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" volumeName="kubernetes.io/empty-dir/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-tmp" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667588 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962bd32-3fc5-4a46-ba44-0d08701a6cd0" volumeName="kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 22:08:00.667596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667601 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962bd32-3fc5-4a46-ba44-0d08701a6cd0" volumeName="kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667624 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e40ade3f-f022-48c6-b9ed-a219100b4ff2" volumeName="kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-script-lib" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667664 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec96604b-256b-404b-95d1-5fe5cb00a147" volumeName="kubernetes.io/projected/ec96604b-256b-404b-95d1-5fe5cb00a147-kube-api-access-nclln" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667682 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b9a551-4faa-46fb-8d85-9a39e39d9191" volumeName="kubernetes.io/projected/03b9a551-4faa-46fb-8d85-9a39e39d9191-kube-api-access-qxdcf" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667695 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23923c80-00d0-4e1b-adbc-89cc9a992709" volumeName="kubernetes.io/secret/23923c80-00d0-4e1b-adbc-89cc9a992709-agent-certs" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667707 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433b8aa2-23d3-432b-abf6-15b7b334b287" volumeName="kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667719 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5692356c-6609-46d4-9cd7-513a0d883779" volumeName="kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-client-ca-bundle" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667730 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5692356c-6609-46d4-9cd7-513a0d883779" volumeName="kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667742 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0f26ebc-df35-404e-9466-9f5d73478ef7" volumeName="kubernetes.io/configmap/a0f26ebc-df35-404e-9466-9f5d73478ef7-metrics-client-ca" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667756 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e40ade3f-f022-48c6-b9ed-a219100b4ff2" volumeName="kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-env-overrides" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667769 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02608c76-a776-4f06-94d3-eb3527a9dcdf" volumeName="kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-metrics-client-ca" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667780 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433b8aa2-23d3-432b-abf6-15b7b334b287" volumeName="kubernetes.io/empty-dir/433b8aa2-23d3-432b-abf6-15b7b334b287-volume-directive-shadow" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667790 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433b8aa2-23d3-432b-abf6-15b7b334b287" volumeName="kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667801 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="790226ec-d17d-47e7-b32f-da97c9e983c2" volumeName="kubernetes.io/empty-dir/790226ec-d17d-47e7-b32f-da97c9e983c2-data-volume" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667812 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a344707b-bc31-4587-ad1b-5da15d0a4dc9" volumeName="kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-tuned" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667825 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beccabec-c5c9-4f17-a8ba-c23e1254c391" volumeName="kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667837 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beccabec-c5c9-4f17-a8ba-c23e1254c391" volumeName="kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667849 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c056927-9f67-4a21-849b-dd71c606f245" volumeName="kubernetes.io/projected/1c056927-9f67-4a21-849b-dd71c606f245-kube-api-access-xzspk" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667860 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5692356c-6609-46d4-9cd7-513a0d883779" volumeName="kubernetes.io/projected/5692356c-6609-46d4-9cd7-513a0d883779-kube-api-access-npwx8" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667875 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="790226ec-d17d-47e7-b32f-da97c9e983c2" volumeName="kubernetes.io/secret/790226ec-d17d-47e7-b32f-da97c9e983c2-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667888 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962bd32-3fc5-4a46-ba44-0d08701a6cd0" volumeName="kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-grpc-tls" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667902 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed75a83a-9f07-4a2f-be7f-1a0b249728c0" volumeName="kubernetes.io/projected/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-kube-api-access-64bmn" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.667916 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5692356c-6609-46d4-9cd7-513a0d883779" volumeName="kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668061 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c1391f4-b5e9-4e91-88b3-208c56c2a153" volumeName="kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-binary-copy" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668075 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d71ea07-fa84-4b2a-b06d-f7b05d7a126d" volumeName="kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668088 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d71ea07-fa84-4b2a-b06d-f7b05d7a126d" volumeName="kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668102 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" volumeName="kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668114 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87e0c26d-7122-4e83-b875-35a4302cd6e1" volumeName="kubernetes.io/projected/87e0c26d-7122-4e83-b875-35a4302cd6e1-kube-api-access-9x8hb" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668126 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b595e184-27c8-4ffc-b50c-9764f013ef04" volumeName="kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668138 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beccabec-c5c9-4f17-a8ba-c23e1254c391" volumeName="kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668149 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d71ea07-fa84-4b2a-b06d-f7b05d7a126d" volumeName="kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668160 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" volumeName="kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668172 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c201439-de30-4e86-88ca-c0826c25fe86" volumeName="kubernetes.io/projected/6c201439-de30-4e86-88ca-c0826c25fe86-kube-api-access-jk9dv" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668186 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e6d3f43-3683-4c19-8b29-849e581ccd37" volumeName="kubernetes.io/secret/9e6d3f43-3683-4c19-8b29-849e581ccd37-cert" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668202 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beccabec-c5c9-4f17-a8ba-c23e1254c391" volumeName="kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668216 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e40ade3f-f022-48c6-b9ed-a219100b4ff2" volumeName="kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-config" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668228 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02608c76-a776-4f06-94d3-eb3527a9dcdf" volumeName="kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-tls" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668239 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c201439-de30-4e86-88ca-c0826c25fe86" volumeName="kubernetes.io/configmap/6c201439-de30-4e86-88ca-c0826c25fe86-serviceca" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668378 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b9a551-4faa-46fb-8d85-9a39e39d9191" volumeName="kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668395 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e40ade3f-f022-48c6-b9ed-a219100b4ff2" volumeName="kubernetes.io/projected/e40ade3f-f022-48c6-b9ed-a219100b4ff2-kube-api-access-ggwr4" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668407 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd82c8e7-5bc3-4cb7-be68-a519c025f61f" volumeName="kubernetes.io/configmap/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-iptables-alerter-script" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668417 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd82c8e7-5bc3-4cb7-be68-a519c025f61f" volumeName="kubernetes.io/projected/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-kube-api-access-qlgzg" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668429 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02608c76-a776-4f06-94d3-eb3527a9dcdf" volumeName="kubernetes.io/projected/02608c76-a776-4f06-94d3-eb3527a9dcdf-kube-api-access-fg2lc" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668440 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433b8aa2-23d3-432b-abf6-15b7b334b287" volumeName="kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668451 6234 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:08:00.669046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.668458 6234 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:08:00.695367 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.695314 6234 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:08:00.696977 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.696951 6234 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:08:00.696977 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.696981 6234 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:08:00.697154 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.697004 6234 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:08:00.697154 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:00.697043 6234 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:08:00.718022 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.717988 6234 generic.go:356] "Generic (PLEG): container finished" podID="02608c76-a776-4f06-94d3-eb3527a9dcdf" containerID="b2b80ced960f71cda509e92d6f1b85a8f4ffc508e1c2c2a4c073688a16bc4369" exitCode=0 Nov 03 22:08:00.748807 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.748777 6234 generic.go:356] "Generic (PLEG): container finished" podID="bad3247384fc65111dcedea645af8378" containerID="e666d5967bd143c4b2ac1f278b1c6d082583efd983912bf8e411f92fb70b1e68" exitCode=0 Nov 03 22:08:00.762027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.761988 6234 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="cbf42dee22b994ca265750a738a1c984b805dd8d86de27f7247da829fba84252" exitCode=0 Nov 03 22:08:00.762027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.762027 6234 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="5c694f3808b07888598268d29d297242557f374aea8a58fc17e6901c43776a37" exitCode=0 Nov 03 22:08:00.762212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.762037 6234 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="d842b3ce486706bc98522cae561e522dfd452f5cedd7c53f4d2fc6cb53a425cb" exitCode=0 Nov 03 22:08:00.762212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.762051 6234 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="f42981f8874b8843f23b1c8b24bae33f483d3286ba305e368c72ae7955a619a7" exitCode=0 Nov 03 22:08:00.762212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.762063 6234 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="8423b86d5f760e1a1f5f5ac810b31284257250c41bf8aacbec30792ccc943749" exitCode=0 Nov 03 22:08:00.762212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.762072 6234 generic.go:356] "Generic (PLEG): container finished" podID="5c1391f4-b5e9-4e91-88b3-208c56c2a153" containerID="bc37e8d5fc5b7d8c8f77121b16b8e58442ddb0b9e676b0901ef445471a7c7bc6" exitCode=0 Nov 03 22:08:00.764069 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:00.764001 6234 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:08:00.783799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.783750 6234 manager.go:324] Recovery completed Nov 03 22:08:00.797537 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:00.797489 6234 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 22:08:00.806128 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.806099 6234 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:08:00.808156 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.808137 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:08:00.808253 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.808166 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:08:00.808253 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.808176 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:08:00.808716 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.808695 6234 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:08:00.808794 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.808717 6234 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:08:00.811002 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.810986 6234 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:08:00.811045 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811002 6234 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:08:00.811045 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811005 6234 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:08:00.811045 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811021 6234 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:08:00.811188 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811172 6234 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 22:08:00.811227 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811203 6234 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 22:08:00.811227 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811224 6234 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 22:08:00.811280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811229 6234 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 22:08:00.811280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811235 6234 policy_none.go:50] "Start" Nov 03 22:08:00.811280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811246 6234 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:08:00.811280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811256 6234 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:08:00.811430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811393 6234 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 22:08:00.811430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811402 6234 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 22:08:00.811430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811409 6234 policy_none.go:44] "Start" Nov 03 22:08:00.811925 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811909 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:08:00.812003 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811945 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:08:00.812003 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.811960 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:08:00.816534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.816507 6234 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:08:00.820402 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.820390 6234 manager.go:341] "Starting Device Plugin manager" Nov 03 22:08:00.820455 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:00.820414 6234 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:08:00.820455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.820423 6234 server.go:87] "Starting device plugin registration server" Nov 03 22:08:00.820749 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.820735 6234 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:08:00.821215 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.820751 6234 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:08:00.821309 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.821242 6234 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:08:00.821395 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.821366 6234 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:08:00.821448 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.821403 6234 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:08:00.821448 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.821411 6234 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:08:00.821572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.821449 6234 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:08:00.821392098Z","UUID":"f6b15ee0-5f8f-4af7-ad5f-122e57c9a5ea","Handler":null,"Name":"","Endpoint":""} Nov 03 22:08:00.822752 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.822732 6234 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:08:00.822833 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:00.822752 6234 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:08:00.822833 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:00.822786 6234 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-57.us-west-2.compute.internal\" not found" Nov 03 22:08:00.822833 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.822758 6234 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:08:00.855567 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.855540 6234 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:08:00.921593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.921562 6234 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:08:00.922621 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.922600 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:08:00.922739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.922635 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:08:00.922739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.922646 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-57.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:08:00.922739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.922665 6234 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:00.947891 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.947868 6234 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 22:08:00.948015 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.948000 6234 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:08:00.998217 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.998174 6234 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal"] Nov 03 22:08:00.998899 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.998859 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" event={"ID":"56a5599a8af5a3ab98eedd947c314121","Type":"ContainerStarted","Data":"03770f0f32152907d9f07a0cc90677dba127cfa0f5bd1b16804bff1ef85bd5a0"} Nov 03 22:08:00.998949 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.998919 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" event={"ID":"56a5599a8af5a3ab98eedd947c314121","Type":"ContainerStarted","Data":"0d0c414cba1b5256629dc7cd6b00f18c908e6f4e6d3ffedd9791bcf723e1df8d"} Nov 03 22:08:00.998949 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.998933 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" event={"ID":"bad3247384fc65111dcedea645af8378","Type":"ContainerStarted","Data":"cec161917833f09142d96c90b223b5883bf8537e0736b7a394789028f9121521"} Nov 03 22:08:00.999006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.998948 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" event={"ID":"bad3247384fc65111dcedea645af8378","Type":"ContainerDied","Data":"e666d5967bd143c4b2ac1f278b1c6d082583efd983912bf8e411f92fb70b1e68"} Nov 03 22:08:00.999006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.998886 6234 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:00.999074 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.999006 6234 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:00.999074 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:00.998967 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" event={"ID":"bad3247384fc65111dcedea645af8378","Type":"ContainerStarted","Data":"6bc001e10574ea0052e18de1dcac1a0610084d34cee3c7266a718828f1b0fcda"} Nov 03 22:08:01.048261 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.048202 6234 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.048261 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.048263 6234 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.059596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.059574 6234 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:08:01.059682 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.059612 6234 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.060124 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.060110 6234 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:08:01.060165 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.060151 6234 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.148996 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.148963 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.148996 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.148997 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56a5599a8af5a3ab98eedd947c314121-config\") pod \"kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"56a5599a8af5a3ab98eedd947c314121\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.149209 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.149016 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.250975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.250932 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.250975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.250943 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.251176 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.251003 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.251176 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.251040 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56a5599a8af5a3ab98eedd947c314121-config\") pod \"kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"56a5599a8af5a3ab98eedd947c314121\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.251176 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.251095 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56a5599a8af5a3ab98eedd947c314121-config\") pod \"kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"56a5599a8af5a3ab98eedd947c314121\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.251176 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.251114 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bad3247384fc65111dcedea645af8378-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" (UID: \"bad3247384fc65111dcedea645af8378\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.650019 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.649990 6234 apiserver.go:52] "Watching apiserver" Nov 03 22:08:01.747404 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.747380 6234 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:08:01.748078 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.748037 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-v6n4q","openshift-monitoring/node-exporter-wbfjj","openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db","openshift-network-diagnostics/network-check-target-9bqcf","openshift-network-operator/iptables-alerter-2swns","openshift-console/console-588668ffc6-29dl9","openshift-monitoring/prometheus-operator-9cd89b74c-wkmds","openshift-dns/node-resolver-d8jgp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal","openshift-monitoring/thanos-querier-8ddd84555-gsxvd","openshift-multus/multus-additional-cni-plugins-v7c9n","openshift-multus/network-metrics-daemon-xkldp","kube-system/global-pull-secret-syncer-lwk4s","openshift-console/console-64cffc769c-fpv4l","openshift-insights/insights-runtime-extractor-pjmrm","kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-8zltr","openshift-cluster-node-tuning-operator/tuned-4s4p6","openshift-dns/dns-default-5ztpx","openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw","openshift-monitoring/metrics-server-779ff6cb48-8rzf9","openshift-ingress-canary/ingress-canary-rz85g","openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f","openshift-multus/multus-796dw","kube-system/konnectivity-agent-x78ps","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj"] Nov 03 22:08:01.765890 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.765865 6234 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.766047 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.765985 6234 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.766560 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.766541 6234 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:08:01.797738 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.797716 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:08:01.797857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.797713 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.797857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.797747 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.799400 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799360 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.799400 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799364 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.799628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799406 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.799628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799413 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.799628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799534 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:08:01.799628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799544 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.799628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799553 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.799628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799557 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.799628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799584 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.799628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799622 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:08:01.799628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799632 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:08:01.800049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799660 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:08:01.800049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799665 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:08:01.800049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799719 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5mdhj\"" Nov 03 22:08:01.800049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799724 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z5g9w\"" Nov 03 22:08:01.800049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799763 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g6ff6\"" Nov 03 22:08:01.800049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799765 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ng2g2\"" Nov 03 22:08:01.800049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799791 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j54w4\"" Nov 03 22:08:01.800049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799871 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8pftz\"" Nov 03 22:08:01.800049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.799998 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:08:01.800445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.800063 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ckrtj\"" Nov 03 22:08:01.800445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.800070 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bkplg\"" Nov 03 22:08:01.800445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.800160 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgskz\"" Nov 03 22:08:01.800445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.800218 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m27qh\"" Nov 03 22:08:01.800445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.800246 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:08:01.800445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.800325 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:08:01.800654 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.800459 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dngqk\"" Nov 03 22:08:01.801275 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.801262 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxpq\"" Nov 03 22:08:01.801345 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.801293 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:08:01.801520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.801482 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:08:01.801579 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.801561 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsv4s\"" Nov 03 22:08:01.802444 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.802432 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:08:01.802613 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.802598 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.802690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.802614 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.802690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.802673 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.802802 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.802701 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:08:01.802859 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.802826 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.802936 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.802924 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.803013 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.802997 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:08:01.803105 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.803055 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.803105 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.803071 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:08:01.803105 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.803086 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:08:01.803258 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.803212 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:08:01.836731 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.836709 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:08:01.836822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.836771 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.836822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.836806 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.842043 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.842025 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.842138 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.842095 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:08:01.842202 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.842141 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.842261 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.842243 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:08:01.844678 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.844653 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:08:01.844798 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.844724 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 22:08:01.844798 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.844730 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:08:01.844895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.844847 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 22:08:01.844895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.844869 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 22:08:01.844895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.844876 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 22:08:01.846150 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.846134 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 22:08:01.850964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.850950 6234 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:08:01.851030 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.850990 6234 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.851542 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.851528 6234 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:08:01.851607 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.851570 6234 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.us-west-2.compute.internal" Nov 03 22:08:01.854832 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.854816 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c056927-9f67-4a21-849b-dd71c606f245-config-volume\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:08:01.854892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.854843 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-lib-modules\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.854892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.854857 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-host\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.854892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.854881 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:01.855013 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.854924 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c201439-de30-4e86-88ca-c0826c25fe86-serviceca\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:08:01.855013 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.854948 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:01.855013 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.854976 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxdcf\" (UniqueName: \"kubernetes.io/projected/03b9a551-4faa-46fb-8d85-9a39e39d9191-kube-api-access-qxdcf\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:08:01.855013 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.854984 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:01.855013 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.855001 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.855267 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.855048 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-netns\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.855267 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.855075 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.355052924 +0000 UTC m=+1.767238941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:01.855267 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.855094 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c056927-9f67-4a21-849b-dd71c606f245-config-volume\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:08:01.855267 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.855176 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.855171 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-system-cni-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.855766 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23923c80-00d0-4e1b-adbc-89cc9a992709-agent-certs\") pod \"konnectivity-agent-x78ps\" (UID: \"23923c80-00d0-4e1b-adbc-89cc9a992709\") " pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.855833 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-modprobe-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.855869 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-textfile\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.855910 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.35588875 +0000 UTC m=+1.768074748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.855943 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-serving-certs-ca-bundle\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.855992 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856021 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2lc\" (UniqueName: \"kubernetes.io/projected/02608c76-a776-4f06-94d3-eb3527a9dcdf-kube-api-access-fg2lc\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856082 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-textfile\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856063 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856191 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23923c80-00d0-4e1b-adbc-89cc9a992709-agent-certs\") pod \"konnectivity-agent-x78ps\" (UID: \"23923c80-00d0-4e1b-adbc-89cc9a992709\") " pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856202 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-netns\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856236 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856286 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.856304 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856316 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c056927-9f67-4a21-849b-dd71c606f245-metrics-tls\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.856360 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.356337543 +0000 UTC m=+1.768523538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-config" not registered Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856377 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzspk\" (UniqueName: \"kubernetes.io/projected/1c056927-9f67-4a21-849b-dd71c606f245-kube-api-access-xzspk\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856389 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c201439-de30-4e86-88ca-c0826c25fe86-serviceca\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856392 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-serving-certs-ca-bundle\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856412 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/790226ec-d17d-47e7-b32f-da97c9e983c2-data-volume\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856573 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c056927-9f67-4a21-849b-dd71c606f245-metrics-tls\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856620 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-tmp-dir\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856656 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c201439-de30-4e86-88ca-c0826c25fe86-host\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856679 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/790226ec-d17d-47e7-b32f-da97c9e983c2-data-volume\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856729 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-tmp-dir\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856738 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856767 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856780 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-accelerators-collector-config\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.856785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856815 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-tmp\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856848 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnp77\" (UniqueName: \"kubernetes.io/projected/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-api-access-nnp77\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856876 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/790226ec-d17d-47e7-b32f-da97c9e983c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856910 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-bin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856942 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-hostroot\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.856974 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nclln\" (UniqueName: \"kubernetes.io/projected/ec96604b-256b-404b-95d1-5fe5cb00a147-kube-api-access-nclln\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857038 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857052 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857095 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857136 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-device-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.857338 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.857402 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.357382382 +0000 UTC m=+1.769568404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857479 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c056927-9f67-4a21-849b-dd71c606f245-tmp-dir\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857544 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857604 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-k8s-cni-cncf-io\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857636 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-sys-fs\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857672 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vp9t\" (UniqueName: \"kubernetes.io/projected/5c1391f4-b5e9-4e91-88b3-208c56c2a153-kube-api-access-9vp9t\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857693 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-accelerators-collector-config\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857714 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857751 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysconfig\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857775 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-tmp\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857802 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beccabec-c5c9-4f17-a8ba-c23e1254c391-tmp\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857829 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-node-log\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857866 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpph\" (UniqueName: \"kubernetes.io/projected/a0f26ebc-df35-404e-9466-9f5d73478ef7-kube-api-access-2cpph\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857892 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64bmn\" (UniqueName: \"kubernetes.io/projected/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-kube-api-access-64bmn\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857922 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-root\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857951 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-metrics-client-ca\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.857981 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-grpc-tls\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858012 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-slash\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858041 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858073 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858104 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-systemd\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858143 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-client-ca-bundle\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858213 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npwx8\" (UniqueName: \"kubernetes.io/projected/5692356c-6609-46d4-9cd7-513a0d883779-kube-api-access-npwx8\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858247 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9dv\" (UniqueName: \"kubernetes.io/projected/6c201439-de30-4e86-88ca-c0826c25fe86-kube-api-access-jk9dv\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858254 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beccabec-c5c9-4f17-a8ba-c23e1254c391-tmp\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858276 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.858304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858319 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbclk\" (UniqueName: \"kubernetes.io/projected/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-kube-api-access-tbclk\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858345 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-tmp\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858354 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858387 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cnibin\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858415 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-tls\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.858441 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.858535 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.358517744 +0000 UTC m=+1.770703753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858697 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.858439 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03b9a551-4faa-46fb-8d85-9a39e39d9191-metrics-certs\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.859065 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.859108 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-socket-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.859163 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-etc-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.859325 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c056927-9f67-4a21-849b-dd71c606f245-tmp-dir\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.859362 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-bin\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.859570 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.859635 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.35959933 +0000 UTC m=+1.771785347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.859827 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02608c76-a776-4f06-94d3-eb3527a9dcdf-metrics-client-ca\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860010 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-metrics-client-ca\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.860060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860044 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-system-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860082 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-etc-kubernetes\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860110 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-ovn\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860136 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860169 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-iptables-alerter-script\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860203 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-cni-binary-copy\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860232 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-client-tls\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860261 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-federate-client-tls\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860286 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scj68\" (UniqueName: \"kubernetes.io/projected/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-kube-api-access-scj68\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860322 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-config\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860352 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-kubernetes\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860381 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860407 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860441 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/433b8aa2-23d3-432b-abf6-15b7b334b287-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860479 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-client-certs\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860527 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860555 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-sys\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860579 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-tuned\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.860849 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860868 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9d22708-0a5b-41de-bd34-78e11212e865-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7zgbj\" (UID: \"c9d22708-0a5b-41de-bd34-78e11212e865\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860893 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/433b8aa2-23d3-432b-abf6-15b7b334b287-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860919 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.860962 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.360940856 +0000 UTC m=+1.773126864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:01.861011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860982 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:08:01.861972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.860676 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-tmp\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:01.861972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.861130 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b28053b-cac3-4543-a09b-25915e6b3f0c-original-pull-secret\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:08:01.861972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.861258 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-tuned\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.861972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.861302 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-metrics-client-ca\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.861972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.861051 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.861972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.861591 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.861972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.861705 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-config\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.862212 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.862121 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:01.862212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862136 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/790226ec-d17d-47e7-b32f-da97c9e983c2-crio-socket\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.862212 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.862181 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.362165611 +0000 UTC m=+1.774351622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:01.862212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862204 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-metrics-client-ca\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.862369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862235 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-tls\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.862369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862266 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-iptables-alerter-script\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:08:01.862369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862261 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.862369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862304 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-socket-dir-parent\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.862369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862327 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-kubelet-config\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:08:01.862564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862388 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-kubelet\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.862564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862415 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-metrics-client-ca\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.862564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862421 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-systemd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.862564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862464 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.862564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862479 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpsm4\" (UniqueName: \"kubernetes.io/projected/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-api-access-bpsm4\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.862564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862524 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlgzg\" (UniqueName: \"kubernetes.io/projected/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-kube-api-access-qlgzg\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:08:01.862564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862551 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f26ebc-df35-404e-9466-9f5d73478ef7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:08:01.862847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862575 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-conf\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.862847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862599 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.862847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862623 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.862847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862649 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23923c80-00d0-4e1b-adbc-89cc9a992709-konnectivity-ca\") pod \"konnectivity-agent-x78ps\" (UID: \"23923c80-00d0-4e1b-adbc-89cc9a992709\") " pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:08:01.862847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862672 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-tmp\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.862847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862624 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-cni-binary-copy\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.862847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862712 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.862847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862773 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f26ebc-df35-404e-9466-9f5d73478ef7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:08:01.862847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862845 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a344707b-bc31-4587-ad1b-5da15d0a4dc9-tmp\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.863216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862844 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.863216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862850 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.863216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862959 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggwr4\" (UniqueName: \"kubernetes.io/projected/e40ade3f-f022-48c6-b9ed-a219100b4ff2-kube-api-access-ggwr4\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.863216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.862968 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23923c80-00d0-4e1b-adbc-89cc9a992709-konnectivity-ca\") pod \"konnectivity-agent-x78ps\" (UID: \"23923c80-00d0-4e1b-adbc-89cc9a992709\") " pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:08:01.863216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863003 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.863216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863023 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-var-lib-kubelet\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.863216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863038 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-conf-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.863216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863053 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.863216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863110 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:01.863216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863145 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrm4\" (UniqueName: \"kubernetes.io/projected/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-kube-api-access-9nrm4\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863308 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863329 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-multus-certs\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863360 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-var-lib-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863388 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863428 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovn-node-metrics-cert\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863455 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5kc\" (UniqueName: \"kubernetes.io/projected/9e6d3f43-3683-4c19-8b29-849e581ccd37-kube-api-access-kz5kc\") pod \"ingress-canary-rz85g\" (UID: \"9e6d3f43-3683-4c19-8b29-849e581ccd37\") " pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863469 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863472 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863526 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-wtmp\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863613 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-run\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863613 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovn-node-metrics-cert\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863642 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-kubelet\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863672 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-registration-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863695 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8hb\" (UniqueName: \"kubernetes.io/projected/87e0c26d-7122-4e83-b875-35a4302cd6e1-kube-api-access-9x8hb\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863811 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863843 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:08:01.863867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863860 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdqr\" (UniqueName: \"kubernetes.io/projected/a344707b-bc31-4587-ad1b-5da15d0a4dc9-kube-api-access-wxdqr\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863884 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-multus\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863922 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863939 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863954 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-systemd-units\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.863982 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6d3f43-3683-4c19-8b29-849e581ccd37-cert\") pod \"ingress-canary-rz85g\" (UID: \"9e6d3f43-3683-4c19-8b29-849e581ccd37\") " pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.864032 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864039 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864044 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-metrics-server-audit-profiles\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864086 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-daemon-config\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864112 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-env-overrides\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.864134 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.364119118 +0000 UTC m=+1.776305118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864154 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-script-lib\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864172 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-tls\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864223 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-cnibin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864252 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864266 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-env-overrides\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864293 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-dbus\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864307 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5692356c-6609-46d4-9cd7-513a0d883779-metrics-server-audit-profiles\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.864335 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864386 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e40ade3f-f022-48c6-b9ed-a219100b4ff2-ovnkube-script-lib\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.864399 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.364386593 +0000 UTC m=+1.776572589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864395 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-netd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864420 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-daemon-config\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864442 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864485 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-host-slash\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864549 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5692356c-6609-46d4-9cd7-513a0d883779-audit-log\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864571 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864586 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-sys\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864598 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c1391f4-b5e9-4e91-88b3-208c56c2a153-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.864598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864602 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864728 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433b8aa2-23d3-432b-abf6-15b7b334b287-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864728 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864757 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864762 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5692356c-6609-46d4-9cd7-513a0d883779-audit-log\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864776 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-log-socket\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864802 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-hosts-file\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.864822 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.864830 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.864829 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.864866 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.364852241 +0000 UTC m=+1.777038253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-config" not registered Nov 03 22:08:01.865465 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.864947 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.36493804 +0000 UTC m=+1.777124040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:01.899715 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.899686 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 22:08:01.900589 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900541 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 22:08:01.900589 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900543 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 22:08:01.900714 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900590 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 22:08:01.900714 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900582 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 22:08:01.900714 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900665 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:08:01.900838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900790 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mrtjf\"" Nov 03 22:08:01.900838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900809 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:08:01.900922 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900851 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 22:08:01.900922 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900881 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 22:08:01.900922 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900902 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 22:08:01.901044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900934 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbwbg\"" Nov 03 22:08:01.901044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900934 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:08:01.901044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900961 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 22:08:01.901044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900985 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4ri5bql2iielr\"" Nov 03 22:08:01.901044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.900994 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 22:08:01.901044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901033 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901118 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-telemeter-client-tls\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901126 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901144 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901185 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901198 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901213 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nq6b6\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901225 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901230 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901202 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2s7p7\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901249 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-khcr8\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901261 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2hth5\"" Nov 03 22:08:01.901268 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901262 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mtmht\"" Nov 03 22:08:01.901671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901293 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pkt2f\"" Nov 03 22:08:01.901671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901301 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2d6mp\"" Nov 03 22:08:01.901671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901296 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 22:08:01.901671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901313 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 22:08:01.901671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901343 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xzmgl\"" Nov 03 22:08:01.901671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901367 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1o2rm9ggtu00i\"" Nov 03 22:08:01.901671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901405 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.901671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901448 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-client-certs\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.901671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.901595 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.902257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.902241 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-federate-client-tls\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.902317 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.902258 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-tls\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.902317 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.902264 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.903423 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.903406 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.903472 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.903419 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.903472 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.903432 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-secret-metrics-server-tls\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.903599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.903502 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.904661 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.904646 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6d3f43-3683-4c19-8b29-849e581ccd37-cert\") pod \"ingress-canary-rz85g\" (UID: \"9e6d3f43-3683-4c19-8b29-849e581ccd37\") " pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:08:01.904710 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.904667 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:08:01.904710 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.904675 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:08:01.904710 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.904650 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-tls\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.905806 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.905790 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.905903 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.905892 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.906685 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.906668 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.907832 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.907817 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/790226ec-d17d-47e7-b32f-da97c9e983c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.909951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.909932 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0f26ebc-df35-404e-9466-9f5d73478ef7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:08:01.910065 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.910052 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-secret-grpc-tls\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.912200 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.912186 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9d22708-0a5b-41de-bd34-78e11212e865-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7zgbj\" (UID: \"c9d22708-0a5b-41de-bd34-78e11212e865\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" Nov 03 22:08:01.912242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.912200 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:08:01.912282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.912239 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5692356c-6609-46d4-9cd7-513a0d883779-client-ca-bundle\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.916538 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.916518 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vp9t\" (UniqueName: \"kubernetes.io/projected/5c1391f4-b5e9-4e91-88b3-208c56c2a153-kube-api-access-9vp9t\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.920788 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.920378 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzspk\" (UniqueName: \"kubernetes.io/projected/1c056927-9f67-4a21-849b-dd71c606f245-kube-api-access-xzspk\") pod \"dns-default-5ztpx\" (UID: \"1c056927-9f67-4a21-849b-dd71c606f245\") " pod="openshift-dns/dns-default-5ztpx" Nov 03 22:08:01.920788 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.920408 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:01.920788 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.920432 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:01.920788 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.920447 6234 projected.go:196] Error preparing data for projected volume kube-api-access-9l9x4 for pod openshift-console/console-588668ffc6-29dl9: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:01.920788 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.920467 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2lc\" (UniqueName: \"kubernetes.io/projected/02608c76-a776-4f06-94d3-eb3527a9dcdf-kube-api-access-fg2lc\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.920788 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.920547 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4 podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.420530624 +0000 UTC m=+1.832716636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9x4" (UniqueName: "kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:01.920788 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.920575 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nclln\" (UniqueName: \"kubernetes.io/projected/ec96604b-256b-404b-95d1-5fe5cb00a147-kube-api-access-nclln\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.920788 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.920737 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npwx8\" (UniqueName: \"kubernetes.io/projected/5692356c-6609-46d4-9cd7-513a0d883779-kube-api-access-npwx8\") pod \"metrics-server-779ff6cb48-8rzf9\" (UID: \"5692356c-6609-46d4-9cd7-513a0d883779\") " pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:01.920788 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.920751 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpph\" (UniqueName: \"kubernetes.io/projected/a0f26ebc-df35-404e-9466-9f5d73478ef7-kube-api-access-2cpph\") pod \"prometheus-operator-9cd89b74c-wkmds\" (UID: \"a0f26ebc-df35-404e-9466-9f5d73478ef7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wkmds" Nov 03 22:08:01.921082 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.920847 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64bmn\" (UniqueName: \"kubernetes.io/projected/ed75a83a-9f07-4a2f-be7f-1a0b249728c0-kube-api-access-64bmn\") pod \"openshift-state-metrics-7bd7489957-qf2db\" (UID: \"ed75a83a-9f07-4a2f-be7f-1a0b249728c0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qf2db" Nov 03 22:08:01.921082 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.920874 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9dv\" (UniqueName: \"kubernetes.io/projected/6c201439-de30-4e86-88ca-c0826c25fe86-kube-api-access-jk9dv\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:08:01.921138 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.921086 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4gz5\" (UniqueName: \"kubernetes.io/projected/b595e184-27c8-4ffc-b50c-9764f013ef04-kube-api-access-w4gz5\") pod \"network-check-target-9bqcf\" (UID: \"b595e184-27c8-4ffc-b50c-9764f013ef04\") " pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:08:01.921138 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.921103 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnp77\" (UniqueName: \"kubernetes.io/projected/433b8aa2-23d3-432b-abf6-15b7b334b287-kube-api-access-nnp77\") pod \"kube-state-metrics-5dc8d7d89b-vwb8n\" (UID: \"433b8aa2-23d3-432b-abf6-15b7b334b287\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vwb8n" Nov 03 22:08:01.922258 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.922241 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxdcf\" (UniqueName: \"kubernetes.io/projected/03b9a551-4faa-46fb-8d85-9a39e39d9191-kube-api-access-qxdcf\") pod \"network-metrics-daemon-xkldp\" (UID: \"03b9a551-4faa-46fb-8d85-9a39e39d9191\") " pod="openshift-multus/network-metrics-daemon-xkldp" Nov 03 22:08:01.923616 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.923599 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdqr\" (UniqueName: \"kubernetes.io/projected/a344707b-bc31-4587-ad1b-5da15d0a4dc9-kube-api-access-wxdqr\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.923673 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.923623 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:01.923673 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.923647 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:01.923673 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.923668 6234 projected.go:196] Error preparing data for projected volume kube-api-access-s9qj7 for pod openshift-console/console-64cffc769c-fpv4l: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:01.923760 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.923683 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8hb\" (UniqueName: \"kubernetes.io/projected/87e0c26d-7122-4e83-b875-35a4302cd6e1-kube-api-access-9x8hb\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.923790 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:01.923771 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7 podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:02.423753985 +0000 UTC m=+1.835940000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s9qj7" (UniqueName: "kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:01.923953 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.923937 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpsm4\" (UniqueName: \"kubernetes.io/projected/790226ec-d17d-47e7-b32f-da97c9e983c2-kube-api-access-bpsm4\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.924506 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.924477 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlgzg\" (UniqueName: \"kubernetes.io/projected/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-kube-api-access-qlgzg\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:08:01.924541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.924490 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrm4\" (UniqueName: \"kubernetes.io/projected/d962bd32-3fc5-4a46-ba44-0d08701a6cd0-kube-api-access-9nrm4\") pod \"thanos-querier-8ddd84555-gsxvd\" (UID: \"d962bd32-3fc5-4a46-ba44-0d08701a6cd0\") " pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:01.924657 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.924640 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scj68\" (UniqueName: \"kubernetes.io/projected/5d71ea07-fa84-4b2a-b06d-f7b05d7a126d-kube-api-access-scj68\") pod \"telemeter-client-6bfb5bb58-vkr9f\" (UID: \"5d71ea07-fa84-4b2a-b06d-f7b05d7a126d\") " pod="openshift-monitoring/telemeter-client-6bfb5bb58-vkr9f" Nov 03 22:08:01.925450 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.925433 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbclk\" (UniqueName: \"kubernetes.io/projected/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-kube-api-access-tbclk\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:08:01.925949 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.925932 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggwr4\" (UniqueName: \"kubernetes.io/projected/e40ade3f-f022-48c6-b9ed-a219100b4ff2-kube-api-access-ggwr4\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.926004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.925970 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5kc\" (UniqueName: \"kubernetes.io/projected/9e6d3f43-3683-4c19-8b29-849e581ccd37-kube-api-access-kz5kc\") pod \"ingress-canary-rz85g\" (UID: \"9e6d3f43-3683-4c19-8b29-849e581ccd37\") " pod="openshift-ingress-canary/ingress-canary-rz85g" Nov 03 22:08:01.965603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965583 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-var-lib-kubelet\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.965699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965609 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-conf-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.965699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965624 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.965699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965650 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-multus-certs\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.965699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965672 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-var-lib-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.965699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965694 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965700 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-var-lib-kubelet\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965711 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-multus-certs\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965700 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-conf-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965716 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-wtmp\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965736 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-var-lib-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965712 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965763 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-run\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965768 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-node-exporter-wtmp\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965778 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965807 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-kubelet\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965822 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-run\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.965873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965854 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-registration-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965889 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-registration-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965922 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-kubelet\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965954 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-multus\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965971 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.965998 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-systemd-units\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966026 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-multus\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966062 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-cnibin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966068 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966091 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-dbus\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966106 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-netd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966133 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-host-slash\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966064 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-systemd-units\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966138 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-cnibin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966168 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd82c8e7-5bc3-4cb7-be68-a519c025f61f-host-slash\") pod \"iptables-alerter-2swns\" (UID: \"fd82c8e7-5bc3-4cb7-be68-a519c025f61f\") " pod="openshift-network-operator/iptables-alerter-2swns" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966171 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-sys\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966178 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-dbus\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966177 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-netd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966206 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-sys\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.966220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966220 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-log-socket\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966237 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-hosts-file\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966241 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-log-socket\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966259 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966275 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-lib-modules\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966287 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-host\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966323 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-netns\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966346 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-system-cni-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966354 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966362 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-host\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966380 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-netns\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966380 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-system-cni-dir\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966393 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13fdc02e-6d8f-4e82-a8a4-9f295b64285d-hosts-file\") pod \"node-resolver-d8jgp\" (UID: \"13fdc02e-6d8f-4e82-a8a4-9f295b64285d\") " pod="openshift-dns/node-resolver-d8jgp" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966396 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-modprobe-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966436 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-lib-modules\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966450 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966467 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-modprobe-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966481 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-netns\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966555 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966620 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c201439-de30-4e86-88ca-c0826c25fe86-host\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966654 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-netns\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966654 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-bin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966679 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-hostroot\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966685 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-var-lib-cni-bin\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966720 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-device-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966736 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c201439-de30-4e86-88ca-c0826c25fe86-host\") pod \"node-ca-v6n4q\" (UID: \"6c201439-de30-4e86-88ca-c0826c25fe86\") " pod="openshift-image-registry/node-ca-v6n4q" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966754 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-device-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966783 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-hostroot\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966826 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-k8s-cni-cncf-io\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966846 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-sys-fs\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966862 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysconfig\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966877 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-node-log\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966892 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-root\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966892 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-host-run-k8s-cni-cncf-io\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966901 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-sys-fs\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966908 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysconfig\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.966887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966909 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-slash\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966935 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02608c76-a776-4f06-94d3-eb3527a9dcdf-root\") pod \"node-exporter-wbfjj\" (UID: \"02608c76-a776-4f06-94d3-eb3527a9dcdf\") " pod="openshift-monitoring/node-exporter-wbfjj" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966934 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-node-log\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966937 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-systemd\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966962 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-systemd\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966969 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966985 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cnibin\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.966935 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-slash\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967025 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-socket-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967053 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c1391f4-b5e9-4e91-88b3-208c56c2a153-cnibin\") pod \"multus-additional-cni-plugins-v7c9n\" (UID: \"5c1391f4-b5e9-4e91-88b3-208c56c2a153\") " pod="openshift-multus/multus-additional-cni-plugins-v7c9n" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967073 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967091 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-etc-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967118 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-bin\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967141 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-system-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967120 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-socket-dir\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967157 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-etc-kubernetes\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967180 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-ovn\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967193 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-system-cni-dir\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967193 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-etc-openvswitch\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967180 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-cni-bin\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967211 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-etc-kubernetes\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967220 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-kubernetes\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967221 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-ovn\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967270 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-kubernetes\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967314 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967338 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-sys\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967368 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/790226ec-d17d-47e7-b32f-da97c9e983c2-crio-socket\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967368 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-sys\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967396 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-socket-dir-parent\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967434 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-d\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967438 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/790226ec-d17d-47e7-b32f-da97c9e983c2-crio-socket\") pod \"insights-runtime-extractor-pjmrm\" (UID: \"790226ec-d17d-47e7-b32f-da97c9e983c2\") " pod="openshift-insights/insights-runtime-extractor-pjmrm" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967447 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec96604b-256b-404b-95d1-5fe5cb00a147-multus-socket-dir-parent\") pod \"multus-796dw\" (UID: \"ec96604b-256b-404b-95d1-5fe5cb00a147\") " pod="openshift-multus/multus-796dw" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967460 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-kubelet-config\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967476 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-kubelet\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967518 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-systemd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967535 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-conf\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967553 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967517 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b28053b-cac3-4543-a09b-25915e6b3f0c-kubelet-config\") pod \"global-pull-secret-syncer-lwk4s\" (UID: \"5b28053b-cac3-4543-a09b-25915e6b3f0c\") " pod="kube-system/global-pull-secret-syncer-lwk4s" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967591 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-run-systemd\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967594 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40ade3f-f022-48c6-b9ed-a219100b4ff2-host-kubelet\") pod \"ovnkube-node-8zltr\" (UID: \"e40ade3f-f022-48c6-b9ed-a219100b4ff2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967613 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87e0c26d-7122-4e83-b875-35a4302cd6e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dtvw\" (UID: \"87e0c26d-7122-4e83-b875-35a4302cd6e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtvw" Nov 03 22:08:01.968060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:01.967662 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a344707b-bc31-4587-ad1b-5da15d0a4dc9-etc-sysctl-conf\") pod \"tuned-4s4p6\" (UID: \"a344707b-bc31-4587-ad1b-5da15d0a4dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-4s4p6" Nov 03 22:08:02.371423 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371392 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:02.371423 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371426 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371446 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371465 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371482 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371530 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371553 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371556 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371571 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371591 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371556 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371626 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371617 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371603834 +0000 UTC m=+2.783789828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371637 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371651 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371662 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371647196 +0000 UTC m=+2.783833194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-config" not registered Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371675 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371668392 +0000 UTC m=+2.783854387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:02.371672 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371686 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371681035 +0000 UTC m=+2.783867031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371702 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371735 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371761 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371768 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371758175 +0000 UTC m=+2.783944176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371775 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371785 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371810 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371793463 +0000 UTC m=+2.783979468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-config" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371812 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371830 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371823143 +0000 UTC m=+2.784009142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371845 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371838772 +0000 UTC m=+2.784024767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371859 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371853933 +0000 UTC m=+2.784039928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371869 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371865011 +0000 UTC m=+2.784051035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371890 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371945 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.371948 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371972 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.371964016 +0000 UTC m=+2.784150042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.371993 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:02.372211 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.372021 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.37201589 +0000 UTC m=+2.784201885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:02.472775 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.472745 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:02.472906 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:02.472879 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:02.472906 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.472892 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:02.472966 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.472912 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:02.472966 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.472921 6234 projected.go:196] Error preparing data for projected volume kube-api-access-s9qj7 for pod openshift-console/console-64cffc769c-fpv4l: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:02.472966 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.472963 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7 podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.472947466 +0000 UTC m=+2.885133484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9qj7" (UniqueName: "kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:02.473071 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.473009 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:02.473071 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.473027 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:02.473071 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.473036 6234 projected.go:196] Error preparing data for projected volume kube-api-access-9l9x4 for pod openshift-console/console-588668ffc6-29dl9: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:02.473153 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:02.473076 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4 podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:03.473063383 +0000 UTC m=+2.885249404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9x4" (UniqueName: "kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:03.288101 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.288074 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:03.382279 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.382241 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:03.382654 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382324 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:03.382654 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.382378 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:03.382654 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.382536 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:03.382654 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.382573 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:03.382654 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382590 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.382570825 +0000 UTC m=+4.794756834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:03.382654 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382637 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:03.382654 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.382643 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:03.382654 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382663 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382698 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.38268217 +0000 UTC m=+4.794868199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-config" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.382682 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382717 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.382706758 +0000 UTC m=+4.794892753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382726 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382759 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.382747729 +0000 UTC m=+4.794933762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.382784 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382791 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.382833 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382853 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.38284521 +0000 UTC m=+4.795031206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382843 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.382886 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382894 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.382883692 +0000 UTC m=+4.795069728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-config" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382922 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.382935 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382940 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382968 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.382955421 +0000 UTC m=+4.795141452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382999 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.382999 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.382988818 +0000 UTC m=+4.795174814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.383032 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.383025202 +0000 UTC m=+4.795211196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.383051 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.383091 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.383109 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.383103783 +0000 UTC m=+4.795289792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.383136 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.383184 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.383189 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.383219 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.383210874 +0000 UTC m=+4.795396889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:03.383486 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.383231 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.383225646 +0000 UTC m=+4.795411641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:03.483600 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.483566 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:03.483751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:03.483701 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:03.483751 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.483716 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:03.483751 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.483735 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:03.483751 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.483746 6234 projected.go:196] Error preparing data for projected volume kube-api-access-s9qj7 for pod openshift-console/console-64cffc769c-fpv4l: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:03.483903 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.483792 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7 podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.483775832 +0000 UTC m=+4.895961830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9qj7" (UniqueName: "kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:03.483903 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.483841 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:03.483903 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.483859 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:03.483903 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.483867 6234 projected.go:196] Error preparing data for projected volume kube-api-access-9l9x4 for pod openshift-console/console-588668ffc6-29dl9: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:03.484041 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:03.483907 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4 podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:05.483896769 +0000 UTC m=+4.896082764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9x4" (UniqueName: "kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:04.657986 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:04.657950 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5ztpx" Nov 03 22:08:05.404873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.404831 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:05.405060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.404879 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:05.405060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.404907 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:05.405060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.404932 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:05.405060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.404958 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:05.405060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.404982 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:05.405060 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.404958 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:05.405060 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405037 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:05.405060 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405053 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:05.405060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.405012 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405054 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405094 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.405069635 +0000 UTC m=+8.817255652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405009 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405017 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.404987 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405158 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.405141111 +0000 UTC m=+8.817327105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-config" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405178 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.405169573 +0000 UTC m=+8.817355569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405197 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.405186713 +0000 UTC m=+8.817372714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405212 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.40520508 +0000 UTC m=+8.817391083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405238 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.405230601 +0000 UTC m=+8.817416597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-config" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405261 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.405251075 +0000 UTC m=+8.817437073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.405303 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.405345 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.405388 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405399 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405429 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.405435 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405440 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.405430031 +0000 UTC m=+8.817616046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405463 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405479 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.40546629 +0000 UTC m=+8.817652313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405484 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.405529 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405542 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.405530986 +0000 UTC m=+8.817716999 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405562 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.405554822 +0000 UTC m=+8.817740816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:05.405593 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405573 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:05.406322 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.405621 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.405610563 +0000 UTC m=+8.817796563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:05.506962 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.506927 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:05.507098 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:05.507025 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:05.507098 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.507073 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:05.507098 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.507093 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:05.507210 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.507105 6234 projected.go:196] Error preparing data for projected volume kube-api-access-9l9x4 for pod openshift-console/console-588668ffc6-29dl9: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:05.507210 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.507114 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:05.507210 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.507127 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:05.507210 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.507135 6234 projected.go:196] Error preparing data for projected volume kube-api-access-s9qj7 for pod openshift-console/console-64cffc769c-fpv4l: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:05.507210 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.507159 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4 podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.507145529 +0000 UTC m=+8.919331524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9x4" (UniqueName: "kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:05.507210 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:05.507173 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7 podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:09.507167719 +0000 UTC m=+8.919353714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9qj7" (UniqueName: "kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:06.174443 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:06.174412 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:06.174818 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:06.174652 6234 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:08:06.204489 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:06.204464 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7zgbj" Nov 03 22:08:06.707152 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:06.707121 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:06.712338 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:06.712310 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8ddd84555-gsxvd" Nov 03 22:08:08.734749 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:08.734719 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:09.454771 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.454733 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:09.454925 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.454779 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:09.454925 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.454851 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:09.454925 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.454881 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:09.454925 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.454900 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.454931 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.454947 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.454930855 +0000 UTC m=+16.867116867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.454975 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.454984 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455002 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455019 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455043 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.455028914 +0000 UTC m=+16.867214923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455059 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455019 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.455092 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:09.455096 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455100 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.455086151 +0000 UTC m=+16.867272165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455119 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.455112607 +0000 UTC m=+16.867298602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455131 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.45512554 +0000 UTC m=+16.867311535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455143 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.455137467 +0000 UTC m=+16.867323468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-config" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455167 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455212 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455240 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.455220905 +0000 UTC m=+16.867406914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.455175 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455254 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.455248456 +0000 UTC m=+16.867434450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.455279 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.455297 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.455316 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.455334 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455380 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455395 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455417 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455427 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.455414115 +0000 UTC m=+16.867600127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-config" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455446 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.455436885 +0000 UTC m=+16.867622904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455416 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455461 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.455453245 +0000 UTC m=+16.867639252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:09.455589 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.455478 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.455467291 +0000 UTC m=+16.867653319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:09.556076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.556042 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:09.556208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:09.556164 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:09.556261 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.556212 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:09.556261 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.556231 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:09.556261 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.556245 6234 projected.go:196] Error preparing data for projected volume kube-api-access-9l9x4 for pod openshift-console/console-588668ffc6-29dl9: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:09.556346 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.556299 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:09.556346 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.556314 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4 podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.55630045 +0000 UTC m=+16.968486458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9x4" (UniqueName: "kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:09.556346 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.556316 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:09.556346 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.556329 6234 projected.go:196] Error preparing data for projected volume kube-api-access-s9qj7 for pod openshift-console/console-64cffc769c-fpv4l: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:09.556461 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:09.556350 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7 podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:17.556344169 +0000 UTC m=+16.968530164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9qj7" (UniqueName: "kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:10.200130 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:10.200090 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:08:12.991518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:12.991470 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zltr" Nov 03 22:08:17.537133 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537100 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:17.537133 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537138 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537158 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537179 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537195 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537246 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537258 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537300 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537289321 +0000 UTC m=+32.949475326 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537315 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537324 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537311877 +0000 UTC m=+32.949497891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537258 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537340 6234 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537362 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537355905 +0000 UTC m=+32.949541961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-config" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537361 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537258 6234 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537262 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537393 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537376042 +0000 UTC m=+32.949562052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-config" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537409 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537429 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537423175 +0000 UTC m=+32.949609225 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537430 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537442 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537436283 +0000 UTC m=+32.949622277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-serving-cert" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537455 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537449645 +0000 UTC m=+32.949635640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537478 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537510 6234 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537522 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537547 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.53753564 +0000 UTC m=+32.949721656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"console-oauth-config" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537555 6234 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537565 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537570 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537607 6234 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537576 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537570397 +0000 UTC m=+32.949756428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"oauth-serving-cert" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537651 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537640208 +0000 UTC m=+32.949826206 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:17.537652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537668 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537659076 +0000 UTC m=+32.949845106 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : object "openshift-console"/"service-ca" not registered Nov 03 22:08:17.538652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.537757 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:17.538652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537820 6234 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:17.538652 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.537857 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.537846951 +0000 UTC m=+32.950032962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : object "openshift-console"/"trusted-ca-bundle" not registered Nov 03 22:08:17.638980 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.638949 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") pod \"console-64cffc769c-fpv4l\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:17.639150 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:17.639069 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") pod \"console-588668ffc6-29dl9\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:17.639150 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.639121 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:17.639150 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.639143 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:17.639315 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.639156 6234 projected.go:196] Error preparing data for projected volume kube-api-access-s9qj7 for pod openshift-console/console-64cffc769c-fpv4l: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:17.639315 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.639169 6234 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 03 22:08:17.639315 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.639181 6234 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 03 22:08:17.639315 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.639190 6234 projected.go:196] Error preparing data for projected volume kube-api-access-9l9x4 for pod openshift-console/console-588668ffc6-29dl9: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:17.639315 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.639229 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7 podName:5e37b4fe-1eec-4bfe-88ee-95cf34d72e48 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.639210761 +0000 UTC m=+33.051396773 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9qj7" (UniqueName: "kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7") pod "console-64cffc769c-fpv4l" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:17.639315 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:17.639250 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4 podName:beccabec-c5c9-4f17-a8ba-c23e1254c391 nodeName:}" failed. No retries permitted until 2026-11-03 22:08:33.639240661 +0000 UTC m=+33.051426659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9x4" (UniqueName: "kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4") pod "console-588668ffc6-29dl9" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 03 22:08:19.046085 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:19.046055 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:26.777745 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:26.777701 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-64cffc769c-fpv4l" podUID="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" containerName="console" containerID="cri-o://2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537" gracePeriod=15 Nov 03 22:08:26.782322 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:26.782270 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-588668ffc6-29dl9" podUID="beccabec-c5c9-4f17-a8ba-c23e1254c391" containerName="console" containerID="cri-o://95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843" gracePeriod=15 Nov 03 22:08:27.058030 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.058005 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:27.076576 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.076553 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:27.135792 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.135761 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") pod \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " Nov 03 22:08:27.135948 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.135808 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") pod \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " Nov 03 22:08:27.135948 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.135848 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") pod \"beccabec-c5c9-4f17-a8ba-c23e1254c391\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " Nov 03 22:08:27.135948 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.135878 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") pod \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " Nov 03 22:08:27.135948 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.135907 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") pod \"beccabec-c5c9-4f17-a8ba-c23e1254c391\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " Nov 03 22:08:27.136129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.135952 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-tmp\") pod \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " Nov 03 22:08:27.136129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.135984 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") pod \"beccabec-c5c9-4f17-a8ba-c23e1254c391\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " Nov 03 22:08:27.136129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136010 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") pod \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " Nov 03 22:08:27.136129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136052 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") pod \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " Nov 03 22:08:27.136129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136082 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") pod \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " Nov 03 22:08:27.136129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136110 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") pod \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\" (UID: \"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48\") " Nov 03 22:08:27.136428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136160 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") pod \"beccabec-c5c9-4f17-a8ba-c23e1254c391\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " Nov 03 22:08:27.136428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136210 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") pod \"beccabec-c5c9-4f17-a8ba-c23e1254c391\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " Nov 03 22:08:27.136428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136238 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beccabec-c5c9-4f17-a8ba-c23e1254c391-tmp\" (UniqueName: \"kubernetes.io/empty-dir/beccabec-c5c9-4f17-a8ba-c23e1254c391-tmp\") pod \"beccabec-c5c9-4f17-a8ba-c23e1254c391\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " Nov 03 22:08:27.136428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136277 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") pod \"beccabec-c5c9-4f17-a8ba-c23e1254c391\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " Nov 03 22:08:27.136428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136310 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") pod \"beccabec-c5c9-4f17-a8ba-c23e1254c391\" (UID: \"beccabec-c5c9-4f17-a8ba-c23e1254c391\") " Nov 03 22:08:27.136428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136307 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-tmp" pod "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:27.136428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136357 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config" pod "beccabec-c5c9-4f17-a8ba-c23e1254c391" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:27.136428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136364 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle" pod "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:27.136428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136400 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca" pod "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:27.136900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136581 6234 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-trusted-ca-bundle\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.136900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136606 6234 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-config\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.136900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136620 6234 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.136900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136476 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca" pod "beccabec-c5c9-4f17-a8ba-c23e1254c391" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:27.136900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136636 6234 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-tmp\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.136900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136711 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert" pod "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:27.136900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.136731 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert" pod "beccabec-c5c9-4f17-a8ba-c23e1254c391" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:27.137243 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.137018 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beccabec-c5c9-4f17-a8ba-c23e1254c391-tmp" pod "beccabec-c5c9-4f17-a8ba-c23e1254c391" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:27.137243 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.137026 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle" pod "beccabec-c5c9-4f17-a8ba-c23e1254c391" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:27.137243 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.137103 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config" pod "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:27.139029 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.139004 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert" pod "beccabec-c5c9-4f17-a8ba-c23e1254c391" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:27.139141 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.139087 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config" pod "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:27.139141 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.139119 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config" pod "beccabec-c5c9-4f17-a8ba-c23e1254c391" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:27.139225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.139145 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7" pod "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48"). InnerVolumeSpecName "kube-api-access-s9qj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:27.139225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.139155 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert" pod "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" (UID: "5e37b4fe-1eec-4bfe-88ee-95cf34d72e48"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:27.139225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.139196 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4" pod "beccabec-c5c9-4f17-a8ba-c23e1254c391" (UID: "beccabec-c5c9-4f17-a8ba-c23e1254c391"). InnerVolumeSpecName "kube-api-access-9l9x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:27.237366 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237333 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l9x4\" (UniqueName: \"kubernetes.io/projected/beccabec-c5c9-4f17-a8ba-c23e1254c391-kube-api-access-9l9x4\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237366 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237360 6234 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-oauth-serving-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237377 6234 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beccabec-c5c9-4f17-a8ba-c23e1254c391-tmp\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237390 6234 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-serving-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237403 6234 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-trusted-ca-bundle\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237414 6234 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-oauth-config\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237427 6234 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beccabec-c5c9-4f17-a8ba-c23e1254c391-console-oauth-config\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237438 6234 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beccabec-c5c9-4f17-a8ba-c23e1254c391-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237451 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9qj7\" (UniqueName: \"kubernetes.io/projected/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-kube-api-access-s9qj7\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237463 6234 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-serving-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237477 6234 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-console-config\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.237591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.237491 6234 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48-oauth-serving-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:27.862794 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.862761 6234 generic.go:356] "Generic (PLEG): container finished" podID="beccabec-c5c9-4f17-a8ba-c23e1254c391" containerID="95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843" exitCode=0 Nov 03 22:08:27.863180 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.862810 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588668ffc6-29dl9" Nov 03 22:08:27.863180 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.862841 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588668ffc6-29dl9" event={"ID":"beccabec-c5c9-4f17-a8ba-c23e1254c391","Type":"ContainerDied","Data":"95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843"} Nov 03 22:08:27.863180 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.862889 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588668ffc6-29dl9" event={"ID":"beccabec-c5c9-4f17-a8ba-c23e1254c391","Type":"ContainerDied","Data":"6820ffc16879a9154d682d03811d75c42159650624ba2f39cffe69f6302210c6"} Nov 03 22:08:27.863180 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.862912 6234 scope.go:122] "RemoveContainer" containerID="95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843" Nov 03 22:08:27.867915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.867755 6234 generic.go:356] "Generic (PLEG): container finished" podID="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" containerID="2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537" exitCode=0 Nov 03 22:08:27.867915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.867808 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cffc769c-fpv4l" event={"ID":"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48","Type":"ContainerDied","Data":"2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537"} Nov 03 22:08:27.867915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.867864 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cffc769c-fpv4l" event={"ID":"5e37b4fe-1eec-4bfe-88ee-95cf34d72e48","Type":"ContainerDied","Data":"4422470e83279867e1ebbb50306c4393786557de2d6685d16207563890000f1b"} Nov 03 22:08:27.868031 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.867935 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cffc769c-fpv4l" Nov 03 22:08:27.874912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.874892 6234 scope.go:122] "RemoveContainer" containerID="95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843" Nov 03 22:08:27.875189 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:27.875168 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843\": container with ID starting with 95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843 not found: ID does not exist" containerID="95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843" Nov 03 22:08:27.875291 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.875200 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843"} err="failed to get container status \"95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843\": rpc error: code = NotFound desc = could not find container \"95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843\": container with ID starting with 95dcb8789965185617adb6b114a810ee24e5a5d7c4f1380816603bf83d1af843 not found: ID does not exist" Nov 03 22:08:27.875291 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.875238 6234 scope.go:122] "RemoveContainer" containerID="2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537" Nov 03 22:08:27.882446 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.882426 6234 scope.go:122] "RemoveContainer" containerID="2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537" Nov 03 22:08:27.882707 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:27.882692 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537\": container with ID starting with 2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537 not found: ID does not exist" containerID="2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537" Nov 03 22:08:27.882759 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:27.882713 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537"} err="failed to get container status \"2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537\": rpc error: code = NotFound desc = could not find container \"2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537\": container with ID starting with 2b13430371dd2c0b95ffb6e8d78ded7b209103ff67a2192b701047b6576be537 not found: ID does not exist" Nov 03 22:08:28.015586 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:28.015558 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-588668ffc6-29dl9"] Nov 03 22:08:28.062058 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:28.062031 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-588668ffc6-29dl9"] Nov 03 22:08:28.120959 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:28.120888 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cffc769c-fpv4l"] Nov 03 22:08:28.124097 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:28.124067 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64cffc769c-fpv4l"] Nov 03 22:08:28.702092 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:28.702057 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" path="/var/lib/kubelet/pods/5e37b4fe-1eec-4bfe-88ee-95cf34d72e48/volumes" Nov 03 22:08:28.702543 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:28.702528 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beccabec-c5c9-4f17-a8ba-c23e1254c391" path="/var/lib/kubelet/pods/beccabec-c5c9-4f17-a8ba-c23e1254c391/volumes" Nov 03 22:08:28.740166 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:28.740129 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:28.743952 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:28.743926 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" Nov 03 22:08:28.994917 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:28.994831 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:08:29.033550 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:29.033522 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x78ps" Nov 03 22:08:30.202702 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:30.202672 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9bqcf" Nov 03 22:08:45.455665 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.455632 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c549574cf-nvt8d"] Nov 03 22:08:45.456147 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.455912 6234 cpu_manager.go:414] "containerMap: removing container" podUID="beccabec-c5c9-4f17-a8ba-c23e1254c391" containerName="console" Nov 03 22:08:45.456147 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.455923 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beccabec-c5c9-4f17-a8ba-c23e1254c391" containerName="console" Nov 03 22:08:45.456147 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.455932 6234 cpu_manager.go:414] "containerMap: removing container" podUID="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" containerName="console" Nov 03 22:08:45.456147 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.455937 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" containerName="console" Nov 03 22:08:45.456147 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.455994 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="beccabec-c5c9-4f17-a8ba-c23e1254c391" containerName="console" Nov 03 22:08:45.456147 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.456001 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e37b4fe-1eec-4bfe-88ee-95cf34d72e48" containerName="console" Nov 03 22:08:45.457679 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.457664 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.507166 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.507135 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 22:08:45.507166 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.507160 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 22:08:45.554150 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.554123 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 22:08:45.554150 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.554144 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 22:08:45.554361 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.554154 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 22:08:45.554361 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.554314 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 22:08:45.554555 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.554542 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lr7tn\"" Nov 03 22:08:45.574401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.574375 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eff0669e-95c6-40f3-8727-ab2ff8b48d26-console-serving-cert\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.574552 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.574411 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-trusted-ca-bundle\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.574552 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.574431 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eff0669e-95c6-40f3-8727-ab2ff8b48d26-console-oauth-config\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.574552 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.574530 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-console-config\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.574694 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.574625 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-service-ca\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.574694 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.574650 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff0669e-95c6-40f3-8727-ab2ff8b48d26-tmp\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.574694 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.574678 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjrv\" (UniqueName: \"kubernetes.io/projected/eff0669e-95c6-40f3-8727-ab2ff8b48d26-kube-api-access-qnjrv\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.574847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.574730 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-oauth-serving-cert\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.648230 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.648199 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 22:08:45.648520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.648475 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 22:08:45.649003 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.648988 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c549574cf-nvt8d"] Nov 03 22:08:45.675725 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.675707 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-console-config\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.675813 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.675734 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-service-ca\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.675813 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.675752 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff0669e-95c6-40f3-8727-ab2ff8b48d26-tmp\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.675813 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.675771 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjrv\" (UniqueName: \"kubernetes.io/projected/eff0669e-95c6-40f3-8727-ab2ff8b48d26-kube-api-access-qnjrv\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.675813 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.675789 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-oauth-serving-cert\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.676014 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.675836 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eff0669e-95c6-40f3-8727-ab2ff8b48d26-console-serving-cert\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.676014 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.675871 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-trusted-ca-bundle\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.676014 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.675896 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eff0669e-95c6-40f3-8727-ab2ff8b48d26-console-oauth-config\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.676254 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.676222 6234 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:08:45.676254 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.676234 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff0669e-95c6-40f3-8727-ab2ff8b48d26-tmp\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.676616 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.676588 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-service-ca\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.676723 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.676636 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-console-config\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.676784 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.676749 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-oauth-serving-cert\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.676895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.676874 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eff0669e-95c6-40f3-8727-ab2ff8b48d26-trusted-ca-bundle\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.678443 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.678422 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eff0669e-95c6-40f3-8727-ab2ff8b48d26-console-serving-cert\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.678562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.678544 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eff0669e-95c6-40f3-8727-ab2ff8b48d26-console-oauth-config\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.729222 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.729163 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjrv\" (UniqueName: \"kubernetes.io/projected/eff0669e-95c6-40f3-8727-ab2ff8b48d26-kube-api-access-qnjrv\") pod \"console-5c549574cf-nvt8d\" (UID: \"eff0669e-95c6-40f3-8727-ab2ff8b48d26\") " pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:45.768449 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:45.768422 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:46.172605 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:46.172529 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c549574cf-nvt8d"] Nov 03 22:08:46.934343 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:46.934304 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c549574cf-nvt8d" event={"ID":"eff0669e-95c6-40f3-8727-ab2ff8b48d26","Type":"ContainerStarted","Data":"8d2b6570e32c32eb43a79cb9413ea37a9685a53a5b298faa7cfb9dfa5a583efd"} Nov 03 22:08:46.934745 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:46.934367 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c549574cf-nvt8d" event={"ID":"eff0669e-95c6-40f3-8727-ab2ff8b48d26","Type":"ContainerStarted","Data":"a0932e6080cdb894bc72e81398830150cc8cdaa77f16bd51eaa0f3843ddffeb9"} Nov 03 22:08:47.043193 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:47.043135 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c549574cf-nvt8d" podStartSLOduration=2.043122999 podStartE2EDuration="2.043122999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:47.042510773 +0000 UTC m=+46.454696784" watchObservedRunningTime="2026-11-03 22:08:47.043122999 +0000 UTC m=+46.455309007" Nov 03 22:08:55.768857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:55.768825 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:55.769400 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:55.769147 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:55.773313 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:55.773288 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:55.971041 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:08:55.971013 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c549574cf-nvt8d" Nov 03 22:08:58.995709 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:08:58.995665 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:09:28.995377 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:09:28.995338 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:09:58.995156 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:09:58.995110 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:10:28.995239 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:10:28.995193 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:10:58.995506 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:10:58.995455 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:11:27.792190 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.792112 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp"] Nov 03 22:11:27.794238 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.794222 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:27.845587 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.845556 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 22:11:27.892254 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.892211 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 22:11:27.892907 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.892888 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp"] Nov 03 22:11:27.893282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.893264 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4"] Nov 03 22:11:27.895690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.895677 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:27.896560 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.896542 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ql597\"" Nov 03 22:11:27.901736 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.901710 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:27.901856 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.901758 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:27.901856 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.901825 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8tlk\" (UniqueName: \"kubernetes.io/projected/a0d8188d-ef2a-42b1-9121-329453ea4f58-kube-api-access-n8tlk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:27.959454 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.959422 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4"] Nov 03 22:11:27.996479 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:27.996457 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-srt9c\"" Nov 03 22:11:28.000514 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.000479 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz"] Nov 03 22:11:28.002866 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.002851 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.003140 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.003056 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:28.003140 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.003109 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:28.003305 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.003184 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8tlk\" (UniqueName: \"kubernetes.io/projected/a0d8188d-ef2a-42b1-9121-329453ea4f58-kube-api-access-n8tlk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:28.003305 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.003244 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2hmq\" (UniqueName: \"kubernetes.io/projected/e9920e00-98c2-49dd-a572-96eb944347b4-kube-api-access-j2hmq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:28.003465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.003333 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:28.003465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.003379 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:28.003577 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.003545 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:28.003613 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.003596 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:28.058409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.058334 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:11:28.058409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.058358 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:11:28.058583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.058559 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:11:28.062633 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.062610 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz"] Nov 03 22:11:28.065641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.065617 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8tlk\" (UniqueName: \"kubernetes.io/projected/a0d8188d-ef2a-42b1-9121-329453ea4f58-kube-api-access-n8tlk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:28.096658 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.096626 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:11:28.096776 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.096689 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:11:28.096824 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.096776 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:11:28.096876 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.096866 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qcrf7\"" Nov 03 22:11:28.104154 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104137 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2hmq\" (UniqueName: \"kubernetes.io/projected/e9920e00-98c2-49dd-a572-96eb944347b4-kube-api-access-j2hmq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:28.104210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104168 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.104210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104192 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.104273 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104217 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.104307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104271 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.104357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104344 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cb3191-b30f-4ee4-9421-0161c75eaf2f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.104394 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104371 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.104433 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104391 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgqlf\" (UniqueName: \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-kube-api-access-xgqlf\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.104469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104446 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:28.104523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104472 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:28.104852 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104832 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:28.104886 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.104849 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:28.107548 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.107533 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:28.159913 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.159767 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2hmq\" (UniqueName: \"kubernetes.io/projected/e9920e00-98c2-49dd-a572-96eb944347b4-kube-api-access-j2hmq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:28.205036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.205006 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.205036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.205041 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.205260 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.205056 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.205260 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.205073 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.205260 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.205097 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cb3191-b30f-4ee4-9421-0161c75eaf2f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.205260 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.205118 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.205260 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.205148 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgqlf\" (UniqueName: \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-kube-api-access-xgqlf\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.205809 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.205787 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.207327 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.207304 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cb3191-b30f-4ee4-9421-0161c75eaf2f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.207447 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.207429 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:28.207532 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.207449 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.207853 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.207830 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.207897 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.207838 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.259924 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.259845 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.260031 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.259999 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgqlf\" (UniqueName: \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-kube-api-access-xgqlf\") pod \"istiod-openshift-gateway-7876bfcb74-8tdpz\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.313019 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.312947 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:28.471460 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.471426 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp"] Nov 03 22:11:28.474480 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:11:28.474448 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d8188d_ef2a_42b1_9121_329453ea4f58.slice/crio-01984f0557645ed8b5fe6ebfc689f09561a6c672eb05f2d57af3d14b44873eae WatchSource:0}: Error finding container 01984f0557645ed8b5fe6ebfc689f09561a6c672eb05f2d57af3d14b44873eae: Status 404 returned error can't find the container with id 01984f0557645ed8b5fe6ebfc689f09561a6c672eb05f2d57af3d14b44873eae Nov 03 22:11:28.476740 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.476719 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:11:28.481109 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.481086 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" event={"ID":"a0d8188d-ef2a-42b1-9121-329453ea4f58","Type":"ContainerStarted","Data":"01984f0557645ed8b5fe6ebfc689f09561a6c672eb05f2d57af3d14b44873eae"} Nov 03 22:11:28.571057 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.570993 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4"] Nov 03 22:11:28.574337 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:11:28.574285 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9920e00_98c2_49dd_a572_96eb944347b4.slice/crio-2fd847821dbef43a614b625abc241740bc1b467440d416ee8e79f2b7b87e8952 WatchSource:0}: Error finding container 2fd847821dbef43a614b625abc241740bc1b467440d416ee8e79f2b7b87e8952: Status 404 returned error can't find the container with id 2fd847821dbef43a614b625abc241740bc1b467440d416ee8e79f2b7b87e8952 Nov 03 22:11:28.672583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:28.672551 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz"] Nov 03 22:11:28.675784 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:11:28.675758 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cb3191_b30f_4ee4_9421_0161c75eaf2f.slice/crio-55af38905145b8b5e198f19debefa9aa91c8ec92e58f585c7a6f2534e51cde9d WatchSource:0}: Error finding container 55af38905145b8b5e198f19debefa9aa91c8ec92e58f585c7a6f2534e51cde9d: Status 404 returned error can't find the container with id 55af38905145b8b5e198f19debefa9aa91c8ec92e58f585c7a6f2534e51cde9d Nov 03 22:11:28.995429 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:11:28.995390 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:11:29.486063 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:29.486015 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" event={"ID":"81cb3191-b30f-4ee4-9421-0161c75eaf2f","Type":"ContainerStarted","Data":"55af38905145b8b5e198f19debefa9aa91c8ec92e58f585c7a6f2534e51cde9d"} Nov 03 22:11:29.487104 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:29.487074 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" event={"ID":"e9920e00-98c2-49dd-a572-96eb944347b4","Type":"ContainerStarted","Data":"2fd847821dbef43a614b625abc241740bc1b467440d416ee8e79f2b7b87e8952"} Nov 03 22:11:34.071487 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:34.071452 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:11:34.071916 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:34.071554 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:11:34.513464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:34.513391 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" event={"ID":"81cb3191-b30f-4ee4-9421-0161c75eaf2f","Type":"ContainerStarted","Data":"e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463"} Nov 03 22:11:34.513674 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:34.513531 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:34.716375 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:34.716314 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" podStartSLOduration=2.32356695 podStartE2EDuration="7.716297558s" totalImagesPullingTime="5.392730608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:34.669137017 +0000 UTC m=+214.081323060" watchObservedRunningTime="2026-11-03 22:11:34.716297558 +0000 UTC m=+214.128483572" Nov 03 22:11:35.520397 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:35.520346 6234 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-8tdpz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:11:35.520874 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:35.520414 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" podUID="81cb3191-b30f-4ee4-9421-0161c75eaf2f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:11:36.529290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:36.524122 6234 generic.go:356] "Generic (PLEG): container finished" podID="a0d8188d-ef2a-42b1-9121-329453ea4f58" containerID="1d7c976159c5262569c26711a954f6006ed064a95f0a58fcad4d165498b85373" exitCode=0 Nov 03 22:11:36.529290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:36.525542 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" event={"ID":"a0d8188d-ef2a-42b1-9121-329453ea4f58","Type":"ContainerDied","Data":"1d7c976159c5262569c26711a954f6006ed064a95f0a58fcad4d165498b85373"} Nov 03 22:11:36.531794 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:36.531766 6234 generic.go:356] "Generic (PLEG): container finished" podID="e9920e00-98c2-49dd-a572-96eb944347b4" containerID="d8a2bc4dc9ec2cd3de32cabe7c86c78d1bb9cb29d5cb9164efc17c16e7520d5b" exitCode=0 Nov 03 22:11:36.531915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:36.531858 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" event={"ID":"e9920e00-98c2-49dd-a572-96eb944347b4","Type":"ContainerDied","Data":"d8a2bc4dc9ec2cd3de32cabe7c86c78d1bb9cb29d5cb9164efc17c16e7520d5b"} Nov 03 22:11:38.519393 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:38.519356 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:11:38.541641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:38.541612 6234 generic.go:356] "Generic (PLEG): container finished" podID="a0d8188d-ef2a-42b1-9121-329453ea4f58" containerID="10a27120ce00aff1f40b90348c15b1639bc245e1820fd50651258c6a0965978f" exitCode=0 Nov 03 22:11:38.541790 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:38.541699 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" event={"ID":"a0d8188d-ef2a-42b1-9121-329453ea4f58","Type":"ContainerDied","Data":"10a27120ce00aff1f40b90348c15b1639bc245e1820fd50651258c6a0965978f"} Nov 03 22:11:40.551530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:40.551470 6234 generic.go:356] "Generic (PLEG): container finished" podID="e9920e00-98c2-49dd-a572-96eb944347b4" containerID="80f8ab6fc9d0e5c28df5b794fa40bd6059f7af467dc27b598fd5307e1a747bbb" exitCode=0 Nov 03 22:11:40.551983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:40.551534 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" event={"ID":"e9920e00-98c2-49dd-a572-96eb944347b4","Type":"ContainerDied","Data":"80f8ab6fc9d0e5c28df5b794fa40bd6059f7af467dc27b598fd5307e1a747bbb"} Nov 03 22:11:46.581259 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:46.581217 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" event={"ID":"a0d8188d-ef2a-42b1-9121-329453ea4f58","Type":"ContainerStarted","Data":"b8740bbd5b826e3e18ab799be8a754cbf1194eb6acc8e13a79c3af59207b2704"} Nov 03 22:11:46.690677 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:46.690620 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" podStartSLOduration=3.753672922 podStartE2EDuration="19.690606523s" totalImagesPullingTime="15.936933601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:46.689826816 +0000 UTC m=+226.102012834" watchObservedRunningTime="2026-11-03 22:11:46.690606523 +0000 UTC m=+226.102792541" Nov 03 22:11:47.586370 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:47.586338 6234 generic.go:356] "Generic (PLEG): container finished" podID="a0d8188d-ef2a-42b1-9121-329453ea4f58" containerID="b8740bbd5b826e3e18ab799be8a754cbf1194eb6acc8e13a79c3af59207b2704" exitCode=0 Nov 03 22:11:47.586814 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:47.586422 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" event={"ID":"a0d8188d-ef2a-42b1-9121-329453ea4f58","Type":"ContainerDied","Data":"b8740bbd5b826e3e18ab799be8a754cbf1194eb6acc8e13a79c3af59207b2704"} Nov 03 22:11:47.588469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:47.588448 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" event={"ID":"e9920e00-98c2-49dd-a572-96eb944347b4","Type":"ContainerStarted","Data":"59dd49d6624b8368eeda8b8e2e1507a9b2a58ede4b5fbd52982149700dc0ee2d"} Nov 03 22:11:47.847305 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:47.847211 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" podStartSLOduration=3.383862923 podStartE2EDuration="20.847197277s" totalImagesPullingTime="17.463334354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:47.846649196 +0000 UTC m=+227.258835213" watchObservedRunningTime="2026-11-03 22:11:47.847197277 +0000 UTC m=+227.259383288" Nov 03 22:11:48.594344 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.594311 6234 generic.go:356] "Generic (PLEG): container finished" podID="e9920e00-98c2-49dd-a572-96eb944347b4" containerID="59dd49d6624b8368eeda8b8e2e1507a9b2a58ede4b5fbd52982149700dc0ee2d" exitCode=0 Nov 03 22:11:48.594769 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.594403 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" event={"ID":"e9920e00-98c2-49dd-a572-96eb944347b4","Type":"ContainerDied","Data":"59dd49d6624b8368eeda8b8e2e1507a9b2a58ede4b5fbd52982149700dc0ee2d"} Nov 03 22:11:48.711817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.711788 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:48.792941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.792915 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-bundle\") pod \"a0d8188d-ef2a-42b1-9121-329453ea4f58\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " Nov 03 22:11:48.792941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.792944 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-util\" (UniqueName: \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-util\") pod \"a0d8188d-ef2a-42b1-9121-329453ea4f58\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " Nov 03 22:11:48.793102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.793008 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d8188d-ef2a-42b1-9121-329453ea4f58-kube-api-access-n8tlk\" (UniqueName: \"kubernetes.io/projected/a0d8188d-ef2a-42b1-9121-329453ea4f58-kube-api-access-n8tlk\") pod \"a0d8188d-ef2a-42b1-9121-329453ea4f58\" (UID: \"a0d8188d-ef2a-42b1-9121-329453ea4f58\") " Nov 03 22:11:48.793477 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.793456 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-bundle" pod "a0d8188d-ef2a-42b1-9121-329453ea4f58" (UID: "a0d8188d-ef2a-42b1-9121-329453ea4f58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:11:48.795346 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.795322 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d8188d-ef2a-42b1-9121-329453ea4f58-kube-api-access-n8tlk" pod "a0d8188d-ef2a-42b1-9121-329453ea4f58" (UID: "a0d8188d-ef2a-42b1-9121-329453ea4f58"). InnerVolumeSpecName "kube-api-access-n8tlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:11:48.798437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.798417 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-util" pod "a0d8188d-ef2a-42b1-9121-329453ea4f58" (UID: "a0d8188d-ef2a-42b1-9121-329453ea4f58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:11:48.893601 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.893510 6234 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-bundle\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:48.893601 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.893541 6234 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0d8188d-ef2a-42b1-9121-329453ea4f58-util\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:48.893601 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:48.893553 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8tlk\" (UniqueName: \"kubernetes.io/projected/a0d8188d-ef2a-42b1-9121-329453ea4f58-kube-api-access-n8tlk\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:49.599247 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.599218 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" Nov 03 22:11:49.599626 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.599247 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mphwp" event={"ID":"a0d8188d-ef2a-42b1-9121-329453ea4f58","Type":"ContainerDied","Data":"01984f0557645ed8b5fe6ebfc689f09561a6c672eb05f2d57af3d14b44873eae"} Nov 03 22:11:49.599626 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.599296 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01984f0557645ed8b5fe6ebfc689f09561a6c672eb05f2d57af3d14b44873eae" Nov 03 22:11:49.729312 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.729287 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:49.800554 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.800529 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-bundle\") pod \"e9920e00-98c2-49dd-a572-96eb944347b4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " Nov 03 22:11:49.800695 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.800572 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-util\" (UniqueName: \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-util\") pod \"e9920e00-98c2-49dd-a572-96eb944347b4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " Nov 03 22:11:49.800695 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.800634 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9920e00-98c2-49dd-a572-96eb944347b4-kube-api-access-j2hmq\" (UniqueName: \"kubernetes.io/projected/e9920e00-98c2-49dd-a572-96eb944347b4-kube-api-access-j2hmq\") pod \"e9920e00-98c2-49dd-a572-96eb944347b4\" (UID: \"e9920e00-98c2-49dd-a572-96eb944347b4\") " Nov 03 22:11:49.800947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.800921 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-bundle" pod "e9920e00-98c2-49dd-a572-96eb944347b4" (UID: "e9920e00-98c2-49dd-a572-96eb944347b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:11:49.802761 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.802741 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9920e00-98c2-49dd-a572-96eb944347b4-kube-api-access-j2hmq" pod "e9920e00-98c2-49dd-a572-96eb944347b4" (UID: "e9920e00-98c2-49dd-a572-96eb944347b4"). InnerVolumeSpecName "kube-api-access-j2hmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:11:49.804811 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.804787 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-util" pod "e9920e00-98c2-49dd-a572-96eb944347b4" (UID: "e9920e00-98c2-49dd-a572-96eb944347b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:11:49.901597 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.901524 6234 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-bundle\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:49.901597 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.901552 6234 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9920e00-98c2-49dd-a572-96eb944347b4-util\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:49.901597 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:49.901562 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2hmq\" (UniqueName: \"kubernetes.io/projected/e9920e00-98c2-49dd-a572-96eb944347b4-kube-api-access-j2hmq\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:50.604422 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:50.604379 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" event={"ID":"e9920e00-98c2-49dd-a572-96eb944347b4","Type":"ContainerDied","Data":"2fd847821dbef43a614b625abc241740bc1b467440d416ee8e79f2b7b87e8952"} Nov 03 22:11:50.604422 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:50.604401 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hcgr4" Nov 03 22:11:50.604422 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:11:50.604429 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fd847821dbef43a614b625abc241740bc1b467440d416ee8e79f2b7b87e8952" Nov 03 22:11:58.994379 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:11:58.994339 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:12:06.392434 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392401 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k"] Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392718 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e9920e00-98c2-49dd-a572-96eb944347b4" containerName="pull" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392729 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9920e00-98c2-49dd-a572-96eb944347b4" containerName="pull" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392745 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e9920e00-98c2-49dd-a572-96eb944347b4" containerName="util" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392751 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9920e00-98c2-49dd-a572-96eb944347b4" containerName="util" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392760 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a0d8188d-ef2a-42b1-9121-329453ea4f58" containerName="extract" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392765 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d8188d-ef2a-42b1-9121-329453ea4f58" containerName="extract" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392773 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a0d8188d-ef2a-42b1-9121-329453ea4f58" containerName="pull" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392777 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d8188d-ef2a-42b1-9121-329453ea4f58" containerName="pull" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392793 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e9920e00-98c2-49dd-a572-96eb944347b4" containerName="extract" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392798 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9920e00-98c2-49dd-a572-96eb944347b4" containerName="extract" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392806 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a0d8188d-ef2a-42b1-9121-329453ea4f58" containerName="util" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392810 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d8188d-ef2a-42b1-9121-329453ea4f58" containerName="util" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392858 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d8188d-ef2a-42b1-9121-329453ea4f58" containerName="extract" Nov 03 22:12:06.392951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.392867 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9920e00-98c2-49dd-a572-96eb944347b4" containerName="extract" Nov 03 22:12:06.397057 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.397037 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.439393 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.439360 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k"] Nov 03 22:12:06.472321 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.472294 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 22:12:06.472458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.472437 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 22:12:06.472458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.472448 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 22:12:06.472458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.472444 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n7ktg\"" Nov 03 22:12:06.472678 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.472529 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 22:12:06.472678 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.472544 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 22:12:06.537707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.537673 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248vv\" (UniqueName: \"kubernetes.io/projected/804542e1-3d3e-41a5-b4c8-3fc52defbd67-kube-api-access-248vv\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.537707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.537709 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/804542e1-3d3e-41a5-b4c8-3fc52defbd67-metrics-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.537962 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.537748 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/804542e1-3d3e-41a5-b4c8-3fc52defbd67-manifests\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.537962 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.537835 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/804542e1-3d3e-41a5-b4c8-3fc52defbd67-tmp\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.537962 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.537949 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/804542e1-3d3e-41a5-b4c8-3fc52defbd67-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.538107 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.537971 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/804542e1-3d3e-41a5-b4c8-3fc52defbd67-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.639346 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.639305 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/804542e1-3d3e-41a5-b4c8-3fc52defbd67-manifests\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.639560 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.639380 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/804542e1-3d3e-41a5-b4c8-3fc52defbd67-tmp\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.639560 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.639445 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/804542e1-3d3e-41a5-b4c8-3fc52defbd67-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.639560 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.639473 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/804542e1-3d3e-41a5-b4c8-3fc52defbd67-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.639560 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.639538 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248vv\" (UniqueName: \"kubernetes.io/projected/804542e1-3d3e-41a5-b4c8-3fc52defbd67-kube-api-access-248vv\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.639827 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.639800 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/804542e1-3d3e-41a5-b4c8-3fc52defbd67-manifests\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.639955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.639837 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/804542e1-3d3e-41a5-b4c8-3fc52defbd67-metrics-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.641797 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.641769 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/804542e1-3d3e-41a5-b4c8-3fc52defbd67-tmp\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.642014 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.641994 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/804542e1-3d3e-41a5-b4c8-3fc52defbd67-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.642120 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.642030 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/804542e1-3d3e-41a5-b4c8-3fc52defbd67-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.642209 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.642189 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/804542e1-3d3e-41a5-b4c8-3fc52defbd67-metrics-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:06.711661 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:06.711628 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248vv\" (UniqueName: \"kubernetes.io/projected/804542e1-3d3e-41a5-b4c8-3fc52defbd67-kube-api-access-248vv\") pod \"opendatahub-operator-controller-manager-55b449f4f4-mjl9k\" (UID: \"804542e1-3d3e-41a5-b4c8-3fc52defbd67\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:07.009093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:07.009011 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:07.530989 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:07.530967 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k"] Nov 03 22:12:07.533621 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:12:07.533587 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804542e1_3d3e_41a5_b4c8_3fc52defbd67.slice/crio-5da1d3a75e0134a9fb1475ec12c079aeef1e303b734440504111d4f204d75b64 WatchSource:0}: Error finding container 5da1d3a75e0134a9fb1475ec12c079aeef1e303b734440504111d4f204d75b64: Status 404 returned error can't find the container with id 5da1d3a75e0134a9fb1475ec12c079aeef1e303b734440504111d4f204d75b64 Nov 03 22:12:07.667466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:07.667430 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" event={"ID":"804542e1-3d3e-41a5-b4c8-3fc52defbd67","Type":"ContainerStarted","Data":"5da1d3a75e0134a9fb1475ec12c079aeef1e303b734440504111d4f204d75b64"} Nov 03 22:12:11.685563 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:11.685526 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" event={"ID":"804542e1-3d3e-41a5-b4c8-3fc52defbd67","Type":"ContainerStarted","Data":"08bd654c3513ceab6929bc5fe6bf057dd5bdeedc57516045a938cce05ed7f2de"} Nov 03 22:12:12.691139 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:12.691096 6234 generic.go:356] "Generic (PLEG): container finished" podID="804542e1-3d3e-41a5-b4c8-3fc52defbd67" containerID="08bd654c3513ceab6929bc5fe6bf057dd5bdeedc57516045a938cce05ed7f2de" exitCode=0 Nov 03 22:12:12.691652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:12.691186 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" event={"ID":"804542e1-3d3e-41a5-b4c8-3fc52defbd67","Type":"ContainerDied","Data":"08bd654c3513ceab6929bc5fe6bf057dd5bdeedc57516045a938cce05ed7f2de"} Nov 03 22:12:13.697694 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:13.697660 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" event={"ID":"804542e1-3d3e-41a5-b4c8-3fc52defbd67","Type":"ContainerStarted","Data":"8d92fe958530d2ea5afe24bee6288439fe1239449800afb40b4511c501686902"} Nov 03 22:12:13.698009 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:13.697788 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:13.917084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:13.916968 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" podStartSLOduration=2.730915995 podStartE2EDuration="7.916952343s" totalImagesPullingTime="4.676891221s" totalInitContainerRuntime="509.145127ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:13.86977149 +0000 UTC m=+253.281957507" watchObservedRunningTime="2026-11-03 22:12:13.916952343 +0000 UTC m=+253.329138360" Nov 03 22:12:24.706890 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:24.706862 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-mjl9k" Nov 03 22:12:29.001196 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:12:29.001162 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:12:40.987586 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:40.987550 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9"] Nov 03 22:12:40.990132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:40.990114 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.023591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.023562 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77bbbc33-ee85-4061-922e-fbc1d70acc83-tls-certs\") pod \"kube-auth-proxy-5dbbd9b584-9pvd9\" (UID: \"77bbbc33-ee85-4061-922e-fbc1d70acc83\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.023721 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.023622 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwrnn\" (UniqueName: \"kubernetes.io/projected/77bbbc33-ee85-4061-922e-fbc1d70acc83-kube-api-access-pwrnn\") pod \"kube-auth-proxy-5dbbd9b584-9pvd9\" (UID: \"77bbbc33-ee85-4061-922e-fbc1d70acc83\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.023771 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.023733 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77bbbc33-ee85-4061-922e-fbc1d70acc83-tmp\") pod \"kube-auth-proxy-5dbbd9b584-9pvd9\" (UID: \"77bbbc33-ee85-4061-922e-fbc1d70acc83\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.085763 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.085729 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 22:12:41.085763 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.085729 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 22:12:41.085969 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.085894 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w8jc2\"" Nov 03 22:12:41.086722 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.086696 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9"] Nov 03 22:12:41.125115 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.125091 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77bbbc33-ee85-4061-922e-fbc1d70acc83-tmp\") pod \"kube-auth-proxy-5dbbd9b584-9pvd9\" (UID: \"77bbbc33-ee85-4061-922e-fbc1d70acc83\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.125249 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.125122 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77bbbc33-ee85-4061-922e-fbc1d70acc83-tls-certs\") pod \"kube-auth-proxy-5dbbd9b584-9pvd9\" (UID: \"77bbbc33-ee85-4061-922e-fbc1d70acc83\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.125249 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.125172 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwrnn\" (UniqueName: \"kubernetes.io/projected/77bbbc33-ee85-4061-922e-fbc1d70acc83-kube-api-access-pwrnn\") pod \"kube-auth-proxy-5dbbd9b584-9pvd9\" (UID: \"77bbbc33-ee85-4061-922e-fbc1d70acc83\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.127440 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.127408 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77bbbc33-ee85-4061-922e-fbc1d70acc83-tmp\") pod \"kube-auth-proxy-5dbbd9b584-9pvd9\" (UID: \"77bbbc33-ee85-4061-922e-fbc1d70acc83\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.127654 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.127637 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77bbbc33-ee85-4061-922e-fbc1d70acc83-tls-certs\") pod \"kube-auth-proxy-5dbbd9b584-9pvd9\" (UID: \"77bbbc33-ee85-4061-922e-fbc1d70acc83\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.181237 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.181200 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwrnn\" (UniqueName: \"kubernetes.io/projected/77bbbc33-ee85-4061-922e-fbc1d70acc83-kube-api-access-pwrnn\") pod \"kube-auth-proxy-5dbbd9b584-9pvd9\" (UID: \"77bbbc33-ee85-4061-922e-fbc1d70acc83\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.300869 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.300798 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" Nov 03 22:12:41.664964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.664892 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9"] Nov 03 22:12:41.669340 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:12:41.669310 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bbbc33_ee85_4061_922e_fbc1d70acc83.slice/crio-2570dc4c0d072cd503d7a3a09b871294f2ec3df03a6b22ce58ad09ae25481e77 WatchSource:0}: Error finding container 2570dc4c0d072cd503d7a3a09b871294f2ec3df03a6b22ce58ad09ae25481e77: Status 404 returned error can't find the container with id 2570dc4c0d072cd503d7a3a09b871294f2ec3df03a6b22ce58ad09ae25481e77 Nov 03 22:12:41.808429 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:41.808398 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" event={"ID":"77bbbc33-ee85-4061-922e-fbc1d70acc83","Type":"ContainerStarted","Data":"2570dc4c0d072cd503d7a3a09b871294f2ec3df03a6b22ce58ad09ae25481e77"} Nov 03 22:12:45.832112 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:45.832076 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" event={"ID":"77bbbc33-ee85-4061-922e-fbc1d70acc83","Type":"ContainerStarted","Data":"83f6108f34bba80cb2e1535704cce80e54cb5c0de7309130bb450e8dd81cd1ea"} Nov 03 22:12:45.942456 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:45.942405 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-9pvd9" podStartSLOduration=2.714695226 podStartE2EDuration="5.942392129s" totalImagesPullingTime="3.227696903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:45.941715881 +0000 UTC m=+285.353901925" watchObservedRunningTime="2026-11-03 22:12:45.942392129 +0000 UTC m=+285.354578143" Nov 03 22:12:47.503547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:47.503513 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jn6kl"] Nov 03 22:12:47.506043 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:47.506025 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jn6kl" Nov 03 22:12:47.559007 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:47.558980 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jn6kl"] Nov 03 22:12:47.560888 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:47.560869 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 22:12:47.582253 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:47.582224 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcz8r\" (UniqueName: \"kubernetes.io/projected/edd26322-17ac-47d0-8f4e-044ae4b973ac-kube-api-access-gcz8r\") pod \"authorino-57f7f8659f-jn6kl\" (UID: \"edd26322-17ac-47d0-8f4e-044ae4b973ac\") " pod="kuadrant-system/authorino-57f7f8659f-jn6kl" Nov 03 22:12:47.604500 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:47.604479 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 22:12:47.604611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:47.604555 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vmmlk\"" Nov 03 22:12:47.682900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:47.682869 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcz8r\" (UniqueName: \"kubernetes.io/projected/edd26322-17ac-47d0-8f4e-044ae4b973ac-kube-api-access-gcz8r\") pod \"authorino-57f7f8659f-jn6kl\" (UID: \"edd26322-17ac-47d0-8f4e-044ae4b973ac\") " pod="kuadrant-system/authorino-57f7f8659f-jn6kl" Nov 03 22:12:47.736259 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:47.736227 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcz8r\" (UniqueName: \"kubernetes.io/projected/edd26322-17ac-47d0-8f4e-044ae4b973ac-kube-api-access-gcz8r\") pod \"authorino-57f7f8659f-jn6kl\" (UID: \"edd26322-17ac-47d0-8f4e-044ae4b973ac\") " pod="kuadrant-system/authorino-57f7f8659f-jn6kl" Nov 03 22:12:47.825379 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:47.825287 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jn6kl" Nov 03 22:12:48.189652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:48.189622 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jn6kl"] Nov 03 22:12:48.192639 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:12:48.192606 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedd26322_17ac_47d0_8f4e_044ae4b973ac.slice/crio-5639a6f5b9857e6890fffcc8d8a4745a8f4bd1f501e12e3b9288b62f39f6f8f5 WatchSource:0}: Error finding container 5639a6f5b9857e6890fffcc8d8a4745a8f4bd1f501e12e3b9288b62f39f6f8f5: Status 404 returned error can't find the container with id 5639a6f5b9857e6890fffcc8d8a4745a8f4bd1f501e12e3b9288b62f39f6f8f5 Nov 03 22:12:48.846278 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:48.846244 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jn6kl" event={"ID":"edd26322-17ac-47d0-8f4e-044ae4b973ac","Type":"ContainerStarted","Data":"5639a6f5b9857e6890fffcc8d8a4745a8f4bd1f501e12e3b9288b62f39f6f8f5"} Nov 03 22:12:50.503333 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.503293 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:12:50.505886 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.505860 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" Nov 03 22:12:50.571377 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.571328 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 22:12:50.613557 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.613520 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:12:50.613736 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.613716 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mhkbs\"" Nov 03 22:12:50.614248 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.614150 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7vjj\" (UniqueName: \"kubernetes.io/projected/f669b176-901b-4865-8735-b04ac3c7dff6-kube-api-access-z7vjj\") pod \"limitador-limitador-7c9b87d574-274zz\" (UID: \"f669b176-901b-4865-8735-b04ac3c7dff6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" Nov 03 22:12:50.614248 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.614199 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f669b176-901b-4865-8735-b04ac3c7dff6-config-file\") pod \"limitador-limitador-7c9b87d574-274zz\" (UID: \"f669b176-901b-4865-8735-b04ac3c7dff6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" Nov 03 22:12:50.716589 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.715655 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7vjj\" (UniqueName: \"kubernetes.io/projected/f669b176-901b-4865-8735-b04ac3c7dff6-kube-api-access-z7vjj\") pod \"limitador-limitador-7c9b87d574-274zz\" (UID: \"f669b176-901b-4865-8735-b04ac3c7dff6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" Nov 03 22:12:50.716589 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.715717 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f669b176-901b-4865-8735-b04ac3c7dff6-config-file\") pod \"limitador-limitador-7c9b87d574-274zz\" (UID: \"f669b176-901b-4865-8735-b04ac3c7dff6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" Nov 03 22:12:50.716589 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.716535 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f669b176-901b-4865-8735-b04ac3c7dff6-config-file\") pod \"limitador-limitador-7c9b87d574-274zz\" (UID: \"f669b176-901b-4865-8735-b04ac3c7dff6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" Nov 03 22:12:50.770993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.770898 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7vjj\" (UniqueName: \"kubernetes.io/projected/f669b176-901b-4865-8735-b04ac3c7dff6-kube-api-access-z7vjj\") pod \"limitador-limitador-7c9b87d574-274zz\" (UID: \"f669b176-901b-4865-8735-b04ac3c7dff6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" Nov 03 22:12:50.822396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.822359 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" Nov 03 22:12:50.841750 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:50.841715 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:12:51.197189 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:51.197159 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:12:51.204520 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:12:51.200698 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf669b176_901b_4865_8735_b04ac3c7dff6.slice/crio-40017075c17d83527db8973022268c85a2244ba5a1f9bfaad4d5ebaa1f6c4152 WatchSource:0}: Error finding container 40017075c17d83527db8973022268c85a2244ba5a1f9bfaad4d5ebaa1f6c4152: Status 404 returned error can't find the container with id 40017075c17d83527db8973022268c85a2244ba5a1f9bfaad4d5ebaa1f6c4152 Nov 03 22:12:51.863666 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:51.863614 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" event={"ID":"f669b176-901b-4865-8735-b04ac3c7dff6","Type":"ContainerStarted","Data":"40017075c17d83527db8973022268c85a2244ba5a1f9bfaad4d5ebaa1f6c4152"} Nov 03 22:12:52.869219 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:52.869185 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jn6kl" event={"ID":"edd26322-17ac-47d0-8f4e-044ae4b973ac","Type":"ContainerStarted","Data":"f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b"} Nov 03 22:12:52.974603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:52.974550 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-jn6kl" podStartSLOduration=2.056224464 podStartE2EDuration="5.974535173s" totalImagesPullingTime="3.918310709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:52.973830254 +0000 UTC m=+292.386016272" watchObservedRunningTime="2026-11-03 22:12:52.974535173 +0000 UTC m=+292.386721187" Nov 03 22:12:53.874401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:53.874363 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" event={"ID":"f669b176-901b-4865-8735-b04ac3c7dff6","Type":"ContainerStarted","Data":"ebe92e4c5f27eb7be313f10d764004fe172badd2c78d5c9e682c665b35e67e51"} Nov 03 22:12:53.991859 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:53.991806 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" podStartSLOduration=1.891248195 podStartE2EDuration="3.991790635s" totalImagesPullingTime="2.10054244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:53.991447513 +0000 UTC m=+293.403633530" watchObservedRunningTime="2026-11-03 22:12:53.991790635 +0000 UTC m=+293.403976654" Nov 03 22:12:54.656238 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:54.656192 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jn6kl"] Nov 03 22:12:54.879343 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:54.879282 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-jn6kl" podUID="edd26322-17ac-47d0-8f4e-044ae4b973ac" containerName="authorino" containerID="cri-o://f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b" gracePeriod=30 Nov 03 22:12:54.879817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:54.879467 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" Nov 03 22:12:55.119747 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.119724 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jn6kl" Nov 03 22:12:55.269169 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.269098 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd26322-17ac-47d0-8f4e-044ae4b973ac-kube-api-access-gcz8r\" (UniqueName: \"kubernetes.io/projected/edd26322-17ac-47d0-8f4e-044ae4b973ac-kube-api-access-gcz8r\") pod \"edd26322-17ac-47d0-8f4e-044ae4b973ac\" (UID: \"edd26322-17ac-47d0-8f4e-044ae4b973ac\") " Nov 03 22:12:55.271199 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.271171 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd26322-17ac-47d0-8f4e-044ae4b973ac-kube-api-access-gcz8r" pod "edd26322-17ac-47d0-8f4e-044ae4b973ac" (UID: "edd26322-17ac-47d0-8f4e-044ae4b973ac"). InnerVolumeSpecName "kube-api-access-gcz8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:12:55.369721 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.369692 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcz8r\" (UniqueName: \"kubernetes.io/projected/edd26322-17ac-47d0-8f4e-044ae4b973ac-kube-api-access-gcz8r\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:55.883984 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.883942 6234 generic.go:356] "Generic (PLEG): container finished" podID="edd26322-17ac-47d0-8f4e-044ae4b973ac" containerID="f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b" exitCode=0 Nov 03 22:12:55.884469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.883993 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jn6kl" Nov 03 22:12:55.884469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.884024 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jn6kl" event={"ID":"edd26322-17ac-47d0-8f4e-044ae4b973ac","Type":"ContainerDied","Data":"f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b"} Nov 03 22:12:55.884469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.884064 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jn6kl" event={"ID":"edd26322-17ac-47d0-8f4e-044ae4b973ac","Type":"ContainerDied","Data":"5639a6f5b9857e6890fffcc8d8a4745a8f4bd1f501e12e3b9288b62f39f6f8f5"} Nov 03 22:12:55.884469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.884081 6234 scope.go:122] "RemoveContainer" containerID="f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b" Nov 03 22:12:55.892906 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.892893 6234 scope.go:122] "RemoveContainer" containerID="f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b" Nov 03 22:12:55.893138 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:12:55.893118 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b\": container with ID starting with f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b not found: ID does not exist" containerID="f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b" Nov 03 22:12:55.893183 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:55.893147 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b"} err="failed to get container status \"f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b\": rpc error: code = NotFound desc = could not find container \"f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b\": container with ID starting with f323f105bf092ce60c67cb251ee05810d098e667b330d5b97a3743a605e3c38b not found: ID does not exist" Nov 03 22:12:56.035401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:56.035368 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jn6kl"] Nov 03 22:12:56.036990 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:56.036967 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jn6kl"] Nov 03 22:12:56.703373 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:12:56.703340 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd26322-17ac-47d0-8f4e-044ae4b973ac" path="/var/lib/kubelet/pods/edd26322-17ac-47d0-8f4e-044ae4b973ac/volumes" Nov 03 22:12:58.994477 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:12:58.994438 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:13:01.069913 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.069844 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz"] Nov 03 22:13:01.070311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.070177 6234 cpu_manager.go:414] "containerMap: removing container" podUID="edd26322-17ac-47d0-8f4e-044ae4b973ac" containerName="authorino" Nov 03 22:13:01.070311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.070188 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd26322-17ac-47d0-8f4e-044ae4b973ac" containerName="authorino" Nov 03 22:13:01.070311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.070248 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd26322-17ac-47d0-8f4e-044ae4b973ac" containerName="authorino" Nov 03 22:13:01.074809 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.074790 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.130031 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.130001 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 22:13:01.130217 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.130077 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:13:01.130217 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.130119 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 22:13:01.130217 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.130140 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 22:13:01.177269 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.177244 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 22:13:01.177422 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.177327 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-sczn5\"" Nov 03 22:13:01.178537 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.178518 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz"] Nov 03 22:13:01.212722 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.212696 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386ed88d-a095-4457-86be-84ef3c9e7ce3-metrics-certs\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.212819 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.212730 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rjld\" (UniqueName: \"kubernetes.io/projected/386ed88d-a095-4457-86be-84ef3c9e7ce3-kube-api-access-9rjld\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.212819 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.212754 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/386ed88d-a095-4457-86be-84ef3c9e7ce3-manager-config\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.212901 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.212890 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386ed88d-a095-4457-86be-84ef3c9e7ce3-cert\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.314007 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.313970 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386ed88d-a095-4457-86be-84ef3c9e7ce3-metrics-certs\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.314007 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.314012 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rjld\" (UniqueName: \"kubernetes.io/projected/386ed88d-a095-4457-86be-84ef3c9e7ce3-kube-api-access-9rjld\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.314222 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.314032 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/386ed88d-a095-4457-86be-84ef3c9e7ce3-manager-config\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.314222 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.314096 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386ed88d-a095-4457-86be-84ef3c9e7ce3-cert\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.314808 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.314786 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/386ed88d-a095-4457-86be-84ef3c9e7ce3-manager-config\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.316404 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.316378 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386ed88d-a095-4457-86be-84ef3c9e7ce3-metrics-certs\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.316515 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.316430 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386ed88d-a095-4457-86be-84ef3c9e7ce3-cert\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.367562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.367462 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rjld\" (UniqueName: \"kubernetes.io/projected/386ed88d-a095-4457-86be-84ef3c9e7ce3-kube-api-access-9rjld\") pod \"jobset-controller-manager-5d46b566c9-nwsvz\" (UID: \"386ed88d-a095-4457-86be-84ef3c9e7ce3\") " pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.385993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.385965 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:01.745685 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.745653 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz"] Nov 03 22:13:01.749141 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:13:01.749113 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386ed88d_a095_4457_86be_84ef3c9e7ce3.slice/crio-2db1127195aa7cc59922d26123fecc7623541a75f44a23e4fd2b335837811210 WatchSource:0}: Error finding container 2db1127195aa7cc59922d26123fecc7623541a75f44a23e4fd2b335837811210: Status 404 returned error can't find the container with id 2db1127195aa7cc59922d26123fecc7623541a75f44a23e4fd2b335837811210 Nov 03 22:13:01.910069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:01.910034 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" event={"ID":"386ed88d-a095-4457-86be-84ef3c9e7ce3","Type":"ContainerStarted","Data":"2db1127195aa7cc59922d26123fecc7623541a75f44a23e4fd2b335837811210"} Nov 03 22:13:05.885936 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:05.885908 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-274zz" Nov 03 22:13:10.948480 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:10.948442 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" event={"ID":"386ed88d-a095-4457-86be-84ef3c9e7ce3","Type":"ContainerStarted","Data":"d40356f948a937338ec37d461ceadc9085b929e65728debbbb34a806adb8bba3"} Nov 03 22:13:10.948889 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:10.948555 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:11.055765 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:11.055712 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" podStartSLOduration=1.417361638 podStartE2EDuration="10.055697815s" totalImagesPullingTime="8.638336177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:11.054719802 +0000 UTC m=+310.466905818" watchObservedRunningTime="2026-11-03 22:13:11.055697815 +0000 UTC m=+310.467883825" Nov 03 22:13:21.963142 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:13:21.963111 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5d46b566c9-nwsvz" Nov 03 22:13:28.996010 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:13:28.995970 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:13:58.994823 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:13:58.994782 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:14:14.107056 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.106970 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz"] Nov 03 22:14:14.107513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.107177 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" podUID="81cb3191-b30f-4ee4-9421-0161c75eaf2f" containerName="discovery" containerID="cri-o://e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463" gracePeriod=30 Nov 03 22:14:14.349126 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.349099 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:14:14.439161 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.439136 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-cacerts\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-cacerts\") pod \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " Nov 03 22:14:14.439299 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.439191 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-dns-cert\") pod \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " Nov 03 22:14:14.439299 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.439226 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81cb3191-b30f-4ee4-9421-0161c75eaf2f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cb3191-b30f-4ee4-9421-0161c75eaf2f-local-certs\") pod \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " Nov 03 22:14:14.439299 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.439263 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-token\" (UniqueName: \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-token\") pod \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " Nov 03 22:14:14.439414 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.439310 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-ca-configmap\") pod \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " Nov 03 22:14:14.439462 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.439409 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-kubeconfig\") pod \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " Nov 03 22:14:14.439462 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.439437 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-kube-api-access-xgqlf\" (UniqueName: \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-kube-api-access-xgqlf\") pod \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\" (UID: \"81cb3191-b30f-4ee4-9421-0161c75eaf2f\") " Nov 03 22:14:14.439841 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.439806 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-ca-configmap" pod "81cb3191-b30f-4ee4-9421-0161c75eaf2f" (UID: "81cb3191-b30f-4ee4-9421-0161c75eaf2f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:14.441630 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.441603 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cb3191-b30f-4ee4-9421-0161c75eaf2f-local-certs" pod "81cb3191-b30f-4ee4-9421-0161c75eaf2f" (UID: "81cb3191-b30f-4ee4-9421-0161c75eaf2f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:14:14.441746 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.441636 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-token" pod "81cb3191-b30f-4ee4-9421-0161c75eaf2f" (UID: "81cb3191-b30f-4ee4-9421-0161c75eaf2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:14:14.441921 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.441879 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-kubeconfig" pod "81cb3191-b30f-4ee4-9421-0161c75eaf2f" (UID: "81cb3191-b30f-4ee4-9421-0161c75eaf2f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:14:14.441921 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.441892 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-cacerts" pod "81cb3191-b30f-4ee4-9421-0161c75eaf2f" (UID: "81cb3191-b30f-4ee4-9421-0161c75eaf2f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:14:14.442038 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.441995 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-dns-cert" pod "81cb3191-b30f-4ee4-9421-0161c75eaf2f" (UID: "81cb3191-b30f-4ee4-9421-0161c75eaf2f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:14:14.442230 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.442155 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-kube-api-access-xgqlf" pod "81cb3191-b30f-4ee4-9421-0161c75eaf2f" (UID: "81cb3191-b30f-4ee4-9421-0161c75eaf2f"). InnerVolumeSpecName "kube-api-access-xgqlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:14:14.540789 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.540759 6234 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cb3191-b30f-4ee4-9421-0161c75eaf2f-local-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:14.540789 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.540785 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:14.540789 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.540795 6234 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-ca-configmap\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:14.541012 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.540807 6234 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-kubeconfig\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:14.541012 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.540818 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgqlf\" (UniqueName: \"kubernetes.io/projected/81cb3191-b30f-4ee4-9421-0161c75eaf2f-kube-api-access-xgqlf\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:14.541012 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.540827 6234 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-cacerts\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:14.541012 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:14.540836 6234 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81cb3191-b30f-4ee4-9421-0161c75eaf2f-istio-csr-dns-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:15.215125 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:15.215093 6234 generic.go:356] "Generic (PLEG): container finished" podID="81cb3191-b30f-4ee4-9421-0161c75eaf2f" containerID="e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463" exitCode=0 Nov 03 22:14:15.215605 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:15.215144 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" Nov 03 22:14:15.215605 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:15.215148 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" event={"ID":"81cb3191-b30f-4ee4-9421-0161c75eaf2f","Type":"ContainerDied","Data":"e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463"} Nov 03 22:14:15.215605 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:15.215198 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz" event={"ID":"81cb3191-b30f-4ee4-9421-0161c75eaf2f","Type":"ContainerDied","Data":"55af38905145b8b5e198f19debefa9aa91c8ec92e58f585c7a6f2534e51cde9d"} Nov 03 22:14:15.215605 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:15.215237 6234 scope.go:122] "RemoveContainer" containerID="e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463" Nov 03 22:14:15.225573 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:15.225550 6234 scope.go:122] "RemoveContainer" containerID="e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463" Nov 03 22:14:15.225939 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:14:15.225910 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463\": container with ID starting with e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463 not found: ID does not exist" containerID="e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463" Nov 03 22:14:15.226004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:15.225950 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463"} err="failed to get container status \"e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463\": rpc error: code = NotFound desc = could not find container \"e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463\": container with ID starting with e77cc63875aafb12ccf445d8308ad57458aa6e8c02b43e1631c1cf47b696a463 not found: ID does not exist" Nov 03 22:14:15.415887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:15.415853 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz"] Nov 03 22:14:15.419871 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:15.419847 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8tdpz"] Nov 03 22:14:16.355282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.355250 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx"] Nov 03 22:14:16.355705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.355589 6234 cpu_manager.go:414] "containerMap: removing container" podUID="81cb3191-b30f-4ee4-9421-0161c75eaf2f" containerName="discovery" Nov 03 22:14:16.355705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.355603 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cb3191-b30f-4ee4-9421-0161c75eaf2f" containerName="discovery" Nov 03 22:14:16.355705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.355683 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cb3191-b30f-4ee4-9421-0161c75eaf2f" containerName="discovery" Nov 03 22:14:16.359889 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.359869 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.455791 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.455760 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-maas-api-tls\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.455791 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.455790 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-metrics-certs\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.455978 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.455886 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df23388-6920-47e5-b110-4981bfde2817-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.455978 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.455915 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grjp\" (UniqueName: \"kubernetes.io/projected/9df23388-6920-47e5-b110-4981bfde2817-kube-api-access-7grjp\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.457682 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.457663 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:14:16.457911 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.457880 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:14:16.458036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.457923 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 22:14:16.458036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.457956 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 22:14:16.458036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.458030 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vjv49\"" Nov 03 22:14:16.458918 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.458897 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx"] Nov 03 22:14:16.556367 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.556334 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-maas-api-tls\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.556367 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.556365 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-metrics-certs\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.556643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.556396 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df23388-6920-47e5-b110-4981bfde2817-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.556643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.556428 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grjp\" (UniqueName: \"kubernetes.io/projected/9df23388-6920-47e5-b110-4981bfde2817-kube-api-access-7grjp\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.557152 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.557130 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df23388-6920-47e5-b110-4981bfde2817-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.558851 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.558826 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-metrics-certs\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.558937 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.558894 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-maas-api-tls\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.609395 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.609337 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grjp\" (UniqueName: \"kubernetes.io/projected/9df23388-6920-47e5-b110-4981bfde2817-kube-api-access-7grjp\") pod \"maas-api-6d669c7fcd-xg7zx\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.672257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.672226 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:16.702924 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:16.702897 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cb3191-b30f-4ee4-9421-0161c75eaf2f" path="/var/lib/kubelet/pods/81cb3191-b30f-4ee4-9421-0161c75eaf2f/volumes" Nov 03 22:14:17.036705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:17.036671 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx"] Nov 03 22:14:17.040445 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:14:17.040416 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df23388_6920_47e5_b110_4981bfde2817.slice/crio-2f8556cad4d38e2398ca7a73ce4f67813cba6ebb589415e4cecabdb54a3b09cc WatchSource:0}: Error finding container 2f8556cad4d38e2398ca7a73ce4f67813cba6ebb589415e4cecabdb54a3b09cc: Status 404 returned error can't find the container with id 2f8556cad4d38e2398ca7a73ce4f67813cba6ebb589415e4cecabdb54a3b09cc Nov 03 22:14:17.225442 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:17.225398 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" event={"ID":"9df23388-6920-47e5-b110-4981bfde2817","Type":"ContainerStarted","Data":"2f8556cad4d38e2398ca7a73ce4f67813cba6ebb589415e4cecabdb54a3b09cc"} Nov 03 22:14:21.242755 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:21.242713 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" event={"ID":"9df23388-6920-47e5-b110-4981bfde2817","Type":"ContainerStarted","Data":"d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991"} Nov 03 22:14:21.243130 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:21.242816 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:21.351472 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:21.351421 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" podStartSLOduration=2.152732424 podStartE2EDuration="5.351406755s" totalImagesPullingTime="3.198674331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:21.350683112 +0000 UTC m=+380.762869140" watchObservedRunningTime="2026-11-03 22:14:21.351406755 +0000 UTC m=+380.763592771" Nov 03 22:14:24.320184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:24.320149 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:14:27.254947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:27.254917 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:28.068445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.068411 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-24n6v"] Nov 03 22:14:28.071885 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.071863 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-24n6v" Nov 03 22:14:28.123434 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.123398 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vmmlk\"" Nov 03 22:14:28.124092 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.124069 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-24n6v"] Nov 03 22:14:28.164754 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.164721 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv72z\" (UniqueName: \"kubernetes.io/projected/d9b9d6bf-dd41-4849-ad76-2daab035798e-kube-api-access-rv72z\") pod \"authorino-6f6944bcdd-24n6v\" (UID: \"d9b9d6bf-dd41-4849-ad76-2daab035798e\") " pod="kuadrant-system/authorino-6f6944bcdd-24n6v" Nov 03 22:14:28.250886 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.250856 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq"] Nov 03 22:14:28.254070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.254037 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.265458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.265435 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv72z\" (UniqueName: \"kubernetes.io/projected/d9b9d6bf-dd41-4849-ad76-2daab035798e-kube-api-access-rv72z\") pod \"authorino-6f6944bcdd-24n6v\" (UID: \"d9b9d6bf-dd41-4849-ad76-2daab035798e\") " pod="kuadrant-system/authorino-6f6944bcdd-24n6v" Nov 03 22:14:28.309826 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.309798 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq"] Nov 03 22:14:28.319783 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.319720 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv72z\" (UniqueName: \"kubernetes.io/projected/d9b9d6bf-dd41-4849-ad76-2daab035798e-kube-api-access-rv72z\") pod \"authorino-6f6944bcdd-24n6v\" (UID: \"d9b9d6bf-dd41-4849-ad76-2daab035798e\") " pod="kuadrant-system/authorino-6f6944bcdd-24n6v" Nov 03 22:14:28.366377 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.366344 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2xc9\" (UniqueName: \"kubernetes.io/projected/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-kube-api-access-l2xc9\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.366525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.366384 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-maas-api-tls\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.366525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.366408 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-metrics-certs\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.366604 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.366532 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-openshift-service-ca\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.391005 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.390983 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-24n6v" Nov 03 22:14:28.425311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.425281 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-24n6v"] Nov 03 22:14:28.450273 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.450236 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f449f6747-fvm5n"] Nov 03 22:14:28.454260 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.454236 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f449f6747-fvm5n" Nov 03 22:14:28.467540 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.467516 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2xc9\" (UniqueName: \"kubernetes.io/projected/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-kube-api-access-l2xc9\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.467684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.467560 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-maas-api-tls\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.467684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.467597 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-metrics-certs\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.467684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.467629 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-openshift-service-ca\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.468377 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.468355 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-openshift-service-ca\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.470123 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.470035 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-metrics-certs\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.470123 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.470071 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-maas-api-tls\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.506213 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.506184 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f449f6747-fvm5n"] Nov 03 22:14:28.521680 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.521658 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2xc9\" (UniqueName: \"kubernetes.io/projected/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-kube-api-access-l2xc9\") pod \"maas-api-bb597cf6f-dkjrq\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.566515 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.566472 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:28.568833 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.568813 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nclm\" (UniqueName: \"kubernetes.io/projected/95fb62d9-5fda-467e-81fa-d67b54fa5e36-kube-api-access-7nclm\") pod \"authorino-6f449f6747-fvm5n\" (UID: \"95fb62d9-5fda-467e-81fa-d67b54fa5e36\") " pod="kuadrant-system/authorino-6f449f6747-fvm5n" Nov 03 22:14:28.670279 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.670248 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nclm\" (UniqueName: \"kubernetes.io/projected/95fb62d9-5fda-467e-81fa-d67b54fa5e36-kube-api-access-7nclm\") pod \"authorino-6f449f6747-fvm5n\" (UID: \"95fb62d9-5fda-467e-81fa-d67b54fa5e36\") " pod="kuadrant-system/authorino-6f449f6747-fvm5n" Nov 03 22:14:28.723522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.723471 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nclm\" (UniqueName: \"kubernetes.io/projected/95fb62d9-5fda-467e-81fa-d67b54fa5e36-kube-api-access-7nclm\") pod \"authorino-6f449f6747-fvm5n\" (UID: \"95fb62d9-5fda-467e-81fa-d67b54fa5e36\") " pod="kuadrant-system/authorino-6f449f6747-fvm5n" Nov 03 22:14:28.750668 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.750521 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-24n6v"] Nov 03 22:14:28.753213 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:14:28.753180 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9b9d6bf_dd41_4849_ad76_2daab035798e.slice/crio-981a027f903d39485670208e0521dfa03d351a8eb91b0e09c9dcff8a65b852a1 WatchSource:0}: Error finding container 981a027f903d39485670208e0521dfa03d351a8eb91b0e09c9dcff8a65b852a1: Status 404 returned error can't find the container with id 981a027f903d39485670208e0521dfa03d351a8eb91b0e09c9dcff8a65b852a1 Nov 03 22:14:28.768288 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.768269 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f449f6747-fvm5n" Nov 03 22:14:28.909856 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.909780 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f449f6747-fvm5n"] Nov 03 22:14:28.928873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.928742 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq"] Nov 03 22:14:28.931570 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:14:28.931484 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c8c1a2_4c41_4c58_a7b8_00144d3ae99a.slice/crio-837f9c42fa5ca29c6974dbaa8498ea9b2b829ab58a3d4ca168a091a7e6e61914 WatchSource:0}: Error finding container 837f9c42fa5ca29c6974dbaa8498ea9b2b829ab58a3d4ca168a091a7e6e61914: Status 404 returned error can't find the container with id 837f9c42fa5ca29c6974dbaa8498ea9b2b829ab58a3d4ca168a091a7e6e61914 Nov 03 22:14:28.934877 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.934858 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7478998589-vcrpp"] Nov 03 22:14:28.939474 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.939455 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7478998589-vcrpp" Nov 03 22:14:28.990886 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.990842 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 22:14:28.991748 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:28.991722 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7478998589-vcrpp"] Nov 03 22:14:28.996934 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:14:28.996903 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:14:29.074422 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.074378 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4468t\" (UniqueName: \"kubernetes.io/projected/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-kube-api-access-4468t\") pod \"authorino-7478998589-vcrpp\" (UID: \"ab80cc47-9e3e-4d3b-9220-83134c8f38a9\") " pod="kuadrant-system/authorino-7478998589-vcrpp" Nov 03 22:14:29.074584 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.074456 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-tls-cert\") pod \"authorino-7478998589-vcrpp\" (UID: \"ab80cc47-9e3e-4d3b-9220-83134c8f38a9\") " pod="kuadrant-system/authorino-7478998589-vcrpp" Nov 03 22:14:29.126344 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.126309 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f449f6747-fvm5n"] Nov 03 22:14:29.129722 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:14:29.129691 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95fb62d9_5fda_467e_81fa_d67b54fa5e36.slice/crio-91a8382b26fc95b95edfd90a080ec874e8ce1d7244ed1e066780469f834db195 WatchSource:0}: Error finding container 91a8382b26fc95b95edfd90a080ec874e8ce1d7244ed1e066780469f834db195: Status 404 returned error can't find the container with id 91a8382b26fc95b95edfd90a080ec874e8ce1d7244ed1e066780469f834db195 Nov 03 22:14:29.175461 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.175385 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4468t\" (UniqueName: \"kubernetes.io/projected/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-kube-api-access-4468t\") pod \"authorino-7478998589-vcrpp\" (UID: \"ab80cc47-9e3e-4d3b-9220-83134c8f38a9\") " pod="kuadrant-system/authorino-7478998589-vcrpp" Nov 03 22:14:29.175461 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.175441 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-tls-cert\") pod \"authorino-7478998589-vcrpp\" (UID: \"ab80cc47-9e3e-4d3b-9220-83134c8f38a9\") " pod="kuadrant-system/authorino-7478998589-vcrpp" Nov 03 22:14:29.177764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.177746 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-tls-cert\") pod \"authorino-7478998589-vcrpp\" (UID: \"ab80cc47-9e3e-4d3b-9220-83134c8f38a9\") " pod="kuadrant-system/authorino-7478998589-vcrpp" Nov 03 22:14:29.228849 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.228825 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4468t\" (UniqueName: \"kubernetes.io/projected/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-kube-api-access-4468t\") pod \"authorino-7478998589-vcrpp\" (UID: \"ab80cc47-9e3e-4d3b-9220-83134c8f38a9\") " pod="kuadrant-system/authorino-7478998589-vcrpp" Nov 03 22:14:29.251105 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.251070 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7478998589-vcrpp" Nov 03 22:14:29.279225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.279195 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f449f6747-fvm5n" event={"ID":"95fb62d9-5fda-467e-81fa-d67b54fa5e36","Type":"ContainerStarted","Data":"91a8382b26fc95b95edfd90a080ec874e8ce1d7244ed1e066780469f834db195"} Nov 03 22:14:29.280534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.280489 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-24n6v" event={"ID":"d9b9d6bf-dd41-4849-ad76-2daab035798e","Type":"ContainerStarted","Data":"981a027f903d39485670208e0521dfa03d351a8eb91b0e09c9dcff8a65b852a1"} Nov 03 22:14:29.281691 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.281666 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" event={"ID":"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a","Type":"ContainerStarted","Data":"837f9c42fa5ca29c6974dbaa8498ea9b2b829ab58a3d4ca168a091a7e6e61914"} Nov 03 22:14:29.611390 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:29.611362 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7478998589-vcrpp"] Nov 03 22:14:29.614445 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:14:29.614407 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab80cc47_9e3e_4d3b_9220_83134c8f38a9.slice/crio-5acd808922c41ce70e65f482802978cce5558d33f5b892d4d981c8935e91bd04 WatchSource:0}: Error finding container 5acd808922c41ce70e65f482802978cce5558d33f5b892d4d981c8935e91bd04: Status 404 returned error can't find the container with id 5acd808922c41ce70e65f482802978cce5558d33f5b892d4d981c8935e91bd04 Nov 03 22:14:30.287660 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:30.287628 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" event={"ID":"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a","Type":"ContainerStarted","Data":"a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9"} Nov 03 22:14:30.288130 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:30.287736 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:30.288932 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:30.288912 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7478998589-vcrpp" event={"ID":"ab80cc47-9e3e-4d3b-9220-83134c8f38a9","Type":"ContainerStarted","Data":"5acd808922c41ce70e65f482802978cce5558d33f5b892d4d981c8935e91bd04"} Nov 03 22:14:31.296699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.296644 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7478998589-vcrpp" event={"ID":"ab80cc47-9e3e-4d3b-9220-83134c8f38a9","Type":"ContainerStarted","Data":"36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87"} Nov 03 22:14:31.298445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.298412 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f449f6747-fvm5n" event={"ID":"95fb62d9-5fda-467e-81fa-d67b54fa5e36","Type":"ContainerStarted","Data":"d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75"} Nov 03 22:14:31.298603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.298477 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f449f6747-fvm5n" podUID="95fb62d9-5fda-467e-81fa-d67b54fa5e36" containerName="authorino" containerID="cri-o://d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75" gracePeriod=30 Nov 03 22:14:31.302332 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.301216 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-24n6v" podUID="d9b9d6bf-dd41-4849-ad76-2daab035798e" containerName="authorino" containerID="cri-o://f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496" gracePeriod=30 Nov 03 22:14:31.302332 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.301329 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-24n6v" event={"ID":"d9b9d6bf-dd41-4849-ad76-2daab035798e","Type":"ContainerStarted","Data":"f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496"} Nov 03 22:14:31.404153 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.404104 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7478998589-vcrpp" podStartSLOduration=2.129990976 podStartE2EDuration="3.404089413s" totalImagesPullingTime="1.274098437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:31.403359081 +0000 UTC m=+390.815545100" watchObservedRunningTime="2026-11-03 22:14:31.404089413 +0000 UTC m=+390.816275449" Nov 03 22:14:31.404332 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.404310 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" podStartSLOduration=2.550849732 podStartE2EDuration="3.40430585s" totalImagesPullingTime="853.456118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:30.3983703 +0000 UTC m=+389.810556310" watchObservedRunningTime="2026-11-03 22:14:31.40430585 +0000 UTC m=+390.816491867" Nov 03 22:14:31.512391 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.512042 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f449f6747-fvm5n" podStartSLOduration=2.042373168 podStartE2EDuration="3.5120243s" totalImagesPullingTime="1.469651132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:31.510928735 +0000 UTC m=+390.923114752" watchObservedRunningTime="2026-11-03 22:14:31.5120243 +0000 UTC m=+390.924210317" Nov 03 22:14:31.604601 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.604577 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-24n6v" Nov 03 22:14:31.606606 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.606584 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f449f6747-fvm5n" Nov 03 22:14:31.619637 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.619102 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-24n6v" podStartSLOduration=1.806813048 podStartE2EDuration="3.619084642s" totalImagesPullingTime="1.812271594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:31.617994183 +0000 UTC m=+391.030180201" watchObservedRunningTime="2026-11-03 22:14:31.619084642 +0000 UTC m=+391.031270660" Nov 03 22:14:31.694951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.694911 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95fb62d9-5fda-467e-81fa-d67b54fa5e36-kube-api-access-7nclm\" (UniqueName: \"kubernetes.io/projected/95fb62d9-5fda-467e-81fa-d67b54fa5e36-kube-api-access-7nclm\") pod \"95fb62d9-5fda-467e-81fa-d67b54fa5e36\" (UID: \"95fb62d9-5fda-467e-81fa-d67b54fa5e36\") " Nov 03 22:14:31.695117 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.694966 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b9d6bf-dd41-4849-ad76-2daab035798e-kube-api-access-rv72z\" (UniqueName: \"kubernetes.io/projected/d9b9d6bf-dd41-4849-ad76-2daab035798e-kube-api-access-rv72z\") pod \"d9b9d6bf-dd41-4849-ad76-2daab035798e\" (UID: \"d9b9d6bf-dd41-4849-ad76-2daab035798e\") " Nov 03 22:14:31.697053 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.697020 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b9d6bf-dd41-4849-ad76-2daab035798e-kube-api-access-rv72z" pod "d9b9d6bf-dd41-4849-ad76-2daab035798e" (UID: "d9b9d6bf-dd41-4849-ad76-2daab035798e"). InnerVolumeSpecName "kube-api-access-rv72z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:14:31.697053 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.697046 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95fb62d9-5fda-467e-81fa-d67b54fa5e36-kube-api-access-7nclm" pod "95fb62d9-5fda-467e-81fa-d67b54fa5e36" (UID: "95fb62d9-5fda-467e-81fa-d67b54fa5e36"). InnerVolumeSpecName "kube-api-access-7nclm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:14:31.796623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.796590 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nclm\" (UniqueName: \"kubernetes.io/projected/95fb62d9-5fda-467e-81fa-d67b54fa5e36-kube-api-access-7nclm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:31.796623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:31.796621 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv72z\" (UniqueName: \"kubernetes.io/projected/d9b9d6bf-dd41-4849-ad76-2daab035798e-kube-api-access-rv72z\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:32.305562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.305530 6234 generic.go:356] "Generic (PLEG): container finished" podID="d9b9d6bf-dd41-4849-ad76-2daab035798e" containerID="f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496" exitCode=0 Nov 03 22:14:32.306032 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.305574 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-24n6v" Nov 03 22:14:32.306032 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.305619 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-24n6v" event={"ID":"d9b9d6bf-dd41-4849-ad76-2daab035798e","Type":"ContainerDied","Data":"f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496"} Nov 03 22:14:32.306032 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.305660 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-24n6v" event={"ID":"d9b9d6bf-dd41-4849-ad76-2daab035798e","Type":"ContainerDied","Data":"981a027f903d39485670208e0521dfa03d351a8eb91b0e09c9dcff8a65b852a1"} Nov 03 22:14:32.306032 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.305677 6234 scope.go:122] "RemoveContainer" containerID="f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496" Nov 03 22:14:32.307108 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.307085 6234 generic.go:356] "Generic (PLEG): container finished" podID="95fb62d9-5fda-467e-81fa-d67b54fa5e36" containerID="d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75" exitCode=0 Nov 03 22:14:32.307188 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.307129 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f449f6747-fvm5n" Nov 03 22:14:32.307188 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.307165 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f449f6747-fvm5n" event={"ID":"95fb62d9-5fda-467e-81fa-d67b54fa5e36","Type":"ContainerDied","Data":"d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75"} Nov 03 22:14:32.307275 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.307193 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f449f6747-fvm5n" event={"ID":"95fb62d9-5fda-467e-81fa-d67b54fa5e36","Type":"ContainerDied","Data":"91a8382b26fc95b95edfd90a080ec874e8ce1d7244ed1e066780469f834db195"} Nov 03 22:14:32.315246 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.315233 6234 scope.go:122] "RemoveContainer" containerID="f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496" Nov 03 22:14:32.315663 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:14:32.315474 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496\": container with ID starting with f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496 not found: ID does not exist" containerID="f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496" Nov 03 22:14:32.315663 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.315518 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496"} err="failed to get container status \"f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496\": rpc error: code = NotFound desc = could not find container \"f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496\": container with ID starting with f2779b31454b0b14c25f6eb248402c5fb5fb3cfea92e6d701255d4792f4d6496 not found: ID does not exist" Nov 03 22:14:32.315663 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.315541 6234 scope.go:122] "RemoveContainer" containerID="d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75" Nov 03 22:14:32.323535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.323520 6234 scope.go:122] "RemoveContainer" containerID="d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75" Nov 03 22:14:32.323788 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:14:32.323770 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75\": container with ID starting with d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75 not found: ID does not exist" containerID="d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75" Nov 03 22:14:32.323830 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.323796 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75"} err="failed to get container status \"d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75\": rpc error: code = NotFound desc = could not find container \"d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75\": container with ID starting with d1c57e9da1bd5ad7c13965e2465883ac26d364dc94cbc83337d109458ea80b75 not found: ID does not exist" Nov 03 22:14:32.458036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.458007 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f449f6747-fvm5n"] Nov 03 22:14:32.460155 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.460130 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f449f6747-fvm5n"] Nov 03 22:14:32.562329 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.562240 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-24n6v"] Nov 03 22:14:32.566256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.566226 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-24n6v"] Nov 03 22:14:32.703091 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.703056 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95fb62d9-5fda-467e-81fa-d67b54fa5e36" path="/var/lib/kubelet/pods/95fb62d9-5fda-467e-81fa-d67b54fa5e36/volumes" Nov 03 22:14:32.703413 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:32.703397 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b9d6bf-dd41-4849-ad76-2daab035798e" path="/var/lib/kubelet/pods/d9b9d6bf-dd41-4849-ad76-2daab035798e/volumes" Nov 03 22:14:36.306194 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.306161 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 22:14:36.462330 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.462294 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx"] Nov 03 22:14:36.462582 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.462555 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" podUID="9df23388-6920-47e5-b110-4981bfde2817" containerName="maas-api" containerID="cri-o://d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991" gracePeriod=30 Nov 03 22:14:36.698586 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.698562 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:36.844832 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.844754 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df23388-6920-47e5-b110-4981bfde2817-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df23388-6920-47e5-b110-4981bfde2817-openshift-service-ca\") pod \"9df23388-6920-47e5-b110-4981bfde2817\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " Nov 03 22:14:36.844979 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.844848 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df23388-6920-47e5-b110-4981bfde2817-kube-api-access-7grjp\" (UniqueName: \"kubernetes.io/projected/9df23388-6920-47e5-b110-4981bfde2817-kube-api-access-7grjp\") pod \"9df23388-6920-47e5-b110-4981bfde2817\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " Nov 03 22:14:36.844979 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.844869 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-metrics-certs\") pod \"9df23388-6920-47e5-b110-4981bfde2817\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " Nov 03 22:14:36.844979 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.844925 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-maas-api-tls\") pod \"9df23388-6920-47e5-b110-4981bfde2817\" (UID: \"9df23388-6920-47e5-b110-4981bfde2817\") " Nov 03 22:14:36.845196 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.845165 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df23388-6920-47e5-b110-4981bfde2817-openshift-service-ca" pod "9df23388-6920-47e5-b110-4981bfde2817" (UID: "9df23388-6920-47e5-b110-4981bfde2817"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:14:36.847135 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.847107 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-maas-api-tls" pod "9df23388-6920-47e5-b110-4981bfde2817" (UID: "9df23388-6920-47e5-b110-4981bfde2817"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:14:36.847135 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.847129 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df23388-6920-47e5-b110-4981bfde2817-kube-api-access-7grjp" pod "9df23388-6920-47e5-b110-4981bfde2817" (UID: "9df23388-6920-47e5-b110-4981bfde2817"). InnerVolumeSpecName "kube-api-access-7grjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:14:36.847299 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.847153 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-metrics-certs" pod "9df23388-6920-47e5-b110-4981bfde2817" (UID: "9df23388-6920-47e5-b110-4981bfde2817"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:14:36.945464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.945434 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grjp\" (UniqueName: \"kubernetes.io/projected/9df23388-6920-47e5-b110-4981bfde2817-kube-api-access-7grjp\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:36.945464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.945460 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-metrics-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:36.945464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.945470 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df23388-6920-47e5-b110-4981bfde2817-maas-api-tls\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:36.945691 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:36.945479 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df23388-6920-47e5-b110-4981bfde2817-openshift-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:14:37.331892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:37.331858 6234 generic.go:356] "Generic (PLEG): container finished" podID="9df23388-6920-47e5-b110-4981bfde2817" containerID="d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991" exitCode=0 Nov 03 22:14:37.332321 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:37.331909 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" Nov 03 22:14:37.332321 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:37.331925 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" event={"ID":"9df23388-6920-47e5-b110-4981bfde2817","Type":"ContainerDied","Data":"d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991"} Nov 03 22:14:37.332321 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:37.331967 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx" event={"ID":"9df23388-6920-47e5-b110-4981bfde2817","Type":"ContainerDied","Data":"2f8556cad4d38e2398ca7a73ce4f67813cba6ebb589415e4cecabdb54a3b09cc"} Nov 03 22:14:37.332321 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:37.331987 6234 scope.go:122] "RemoveContainer" containerID="d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991" Nov 03 22:14:37.341318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:37.341302 6234 scope.go:122] "RemoveContainer" containerID="d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991" Nov 03 22:14:37.341671 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:14:37.341652 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991\": container with ID starting with d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991 not found: ID does not exist" containerID="d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991" Nov 03 22:14:37.341742 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:37.341680 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991"} err="failed to get container status \"d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991\": rpc error: code = NotFound desc = could not find container \"d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991\": container with ID starting with d78c95cf2443ff9befac47cc30b31c66826ec22ecc729f6aeb22e0e40175c991 not found: ID does not exist" Nov 03 22:14:37.484449 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:37.484417 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx"] Nov 03 22:14:37.530663 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:37.530633 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xg7zx"] Nov 03 22:14:38.703306 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:14:38.703275 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df23388-6920-47e5-b110-4981bfde2817" path="/var/lib/kubelet/pods/9df23388-6920-47e5-b110-4981bfde2817/volumes" Nov 03 22:14:58.996019 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:14:58.995979 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:15:28.995052 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:15:28.995005 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:15:58.994914 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:15:58.994868 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:16:25.995455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.995421 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft"] Nov 03 22:16:25.995951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.995808 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d9b9d6bf-dd41-4849-ad76-2daab035798e" containerName="authorino" Nov 03 22:16:25.995951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.995822 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b9d6bf-dd41-4849-ad76-2daab035798e" containerName="authorino" Nov 03 22:16:25.995951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.995846 6234 cpu_manager.go:414] "containerMap: removing container" podUID="9df23388-6920-47e5-b110-4981bfde2817" containerName="maas-api" Nov 03 22:16:25.995951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.995852 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df23388-6920-47e5-b110-4981bfde2817" containerName="maas-api" Nov 03 22:16:25.995951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.995870 6234 cpu_manager.go:414] "containerMap: removing container" podUID="95fb62d9-5fda-467e-81fa-d67b54fa5e36" containerName="authorino" Nov 03 22:16:25.995951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.995876 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95fb62d9-5fda-467e-81fa-d67b54fa5e36" containerName="authorino" Nov 03 22:16:25.995951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.995939 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="95fb62d9-5fda-467e-81fa-d67b54fa5e36" containerName="authorino" Nov 03 22:16:25.995951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.995948 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b9d6bf-dd41-4849-ad76-2daab035798e" containerName="authorino" Nov 03 22:16:25.995951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.995956 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df23388-6920-47e5-b110-4981bfde2817" containerName="maas-api" Nov 03 22:16:25.998837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:25.998807 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.048099 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.048075 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 22:16:26.075720 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.075682 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.075900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.075837 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8d8\" (UniqueName: \"kubernetes.io/projected/98ac88a1-8c4e-4fac-b490-d39fe59a433a-kube-api-access-pm8d8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.075900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.075881 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.076006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.075932 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.076006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.075960 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac88a1-8c4e-4fac-b490-d39fe59a433a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.076006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.075987 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.094718 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.094695 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 22:16:26.094834 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.094718 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2wng2\"" Nov 03 22:16:26.094834 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.094718 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 22:16:26.095789 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.095773 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft"] Nov 03 22:16:26.176590 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.176550 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.176753 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.176659 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8d8\" (UniqueName: \"kubernetes.io/projected/98ac88a1-8c4e-4fac-b490-d39fe59a433a-kube-api-access-pm8d8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.176753 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.176692 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.176753 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.176739 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.176895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.176765 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac88a1-8c4e-4fac-b490-d39fe59a433a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.176895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.176788 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.177009 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.176980 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.177072 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.177052 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.177245 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.177227 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.179288 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.179259 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac88a1-8c4e-4fac-b490-d39fe59a433a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.179560 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.179544 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac88a1-8c4e-4fac-b490-d39fe59a433a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.230547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.230522 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8d8\" (UniqueName: \"kubernetes.io/projected/98ac88a1-8c4e-4fac-b490-d39fe59a433a-kube-api-access-pm8d8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nknft\" (UID: \"98ac88a1-8c4e-4fac-b490-d39fe59a433a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.311229 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.311148 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:26.674236 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.674208 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft"] Nov 03 22:16:26.676914 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:16:26.676871 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ac88a1_8c4e_4fac_b490_d39fe59a433a.slice/crio-035c8d8f501a4eaa783d1f5433740ae0d378820c43a4aea5b9369b35ce5b672f WatchSource:0}: Error finding container 035c8d8f501a4eaa783d1f5433740ae0d378820c43a4aea5b9369b35ce5b672f: Status 404 returned error can't find the container with id 035c8d8f501a4eaa783d1f5433740ae0d378820c43a4aea5b9369b35ce5b672f Nov 03 22:16:26.794216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:26.794186 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" event={"ID":"98ac88a1-8c4e-4fac-b490-d39fe59a433a","Type":"ContainerStarted","Data":"035c8d8f501a4eaa783d1f5433740ae0d378820c43a4aea5b9369b35ce5b672f"} Nov 03 22:16:28.806315 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:28.806223 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" event={"ID":"98ac88a1-8c4e-4fac-b490-d39fe59a433a","Type":"ContainerStarted","Data":"e3fafe46b1867c2719b516bb5b7c4c97e351776e4c43a0631031932c65e68b6c"} Nov 03 22:16:28.913890 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:28.913833 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" podStartSLOduration=2.078356334 podStartE2EDuration="3.913819205s" totalImagesPullingTime="1.835462871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:28.912749378 +0000 UTC m=+508.324935395" watchObservedRunningTime="2026-11-03 22:16:28.913819205 +0000 UTC m=+508.326005222" Nov 03 22:16:28.995080 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:16:28.995047 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:16:29.311431 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:29.311397 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:29.625856 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:29.625774 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:16:30.222884 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.222849 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8"] Nov 03 22:16:30.226261 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.226239 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.304409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.304375 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgd2t\" (UniqueName: \"kubernetes.io/projected/48fcdc24-884a-482b-80cb-790ff881816d-kube-api-access-bgd2t\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.304614 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.304442 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fcdc24-884a-482b-80cb-790ff881816d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.304614 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.304477 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.304614 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.304529 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.304614 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.304605 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.304797 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.304635 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.321401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.321374 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 22:16:30.322355 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.322334 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8"] Nov 03 22:16:30.405615 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.405583 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.405780 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.405639 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgd2t\" (UniqueName: \"kubernetes.io/projected/48fcdc24-884a-482b-80cb-790ff881816d-kube-api-access-bgd2t\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.405780 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.405684 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fcdc24-884a-482b-80cb-790ff881816d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.405780 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.405708 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.405780 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.405725 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.405780 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.405766 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.406194 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.406164 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.406299 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.406187 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.406299 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.406205 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.407878 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.407857 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fcdc24-884a-482b-80cb-790ff881816d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.408127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.408111 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fcdc24-884a-482b-80cb-790ff881816d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.459025 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.458992 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgd2t\" (UniqueName: \"kubernetes.io/projected/48fcdc24-884a-482b-80cb-790ff881816d-kube-api-access-bgd2t\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8\" (UID: \"48fcdc24-884a-482b-80cb-790ff881816d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.538012 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.537926 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:30.900183 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.900102 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8"] Nov 03 22:16:30.903287 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:16:30.903257 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48fcdc24_884a_482b_80cb_790ff881816d.slice/crio-f121cc9dceae1179b1335edc9e95cb8db2069aec5092692697251b66419faf26 WatchSource:0}: Error finding container f121cc9dceae1179b1335edc9e95cb8db2069aec5092692697251b66419faf26: Status 404 returned error can't find the container with id f121cc9dceae1179b1335edc9e95cb8db2069aec5092692697251b66419faf26 Nov 03 22:16:30.904648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:30.904629 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:16:31.820203 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:31.820126 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" event={"ID":"48fcdc24-884a-482b-80cb-790ff881816d","Type":"ContainerStarted","Data":"5c68ce61e80a800fc9d5fcb61529f73038938c4abd9c1895c42777fe5cd99d5b"} Nov 03 22:16:31.820203 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:31.820164 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" event={"ID":"48fcdc24-884a-482b-80cb-790ff881816d","Type":"ContainerStarted","Data":"f121cc9dceae1179b1335edc9e95cb8db2069aec5092692697251b66419faf26"} Nov 03 22:16:31.926674 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:31.926625 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podStartSLOduration=1.291187401 podStartE2EDuration="1.926611363s" totalImagesPullingTime="635.423962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:31.926536976 +0000 UTC m=+511.338722993" watchObservedRunningTime="2026-11-03 22:16:31.926611363 +0000 UTC m=+511.338797380" Nov 03 22:16:32.538798 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:32.538762 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:36.311955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.311921 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:36.324886 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.324861 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:36.841049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.841015 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt"] Nov 03 22:16:36.844720 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.844693 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.856194 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.856165 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" Nov 03 22:16:36.859902 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.859873 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.860030 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.859995 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92vtl\" (UniqueName: \"kubernetes.io/projected/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-kube-api-access-92vtl\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.860160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.860145 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.860212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.860191 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.860267 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.860239 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.860323 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.860272 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.893597 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.893563 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 22:16:36.940934 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.940907 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt"] Nov 03 22:16:36.961262 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.961236 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vtl\" (UniqueName: \"kubernetes.io/projected/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-kube-api-access-92vtl\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.961426 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.961293 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.961426 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.961339 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.961426 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.961366 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.961614 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.961433 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.961614 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.961476 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.961821 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.961791 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.961923 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.961839 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.961984 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.961965 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.963758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.963737 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:36.963944 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:36.963929 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:37.013932 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:37.013910 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vtl\" (UniqueName: \"kubernetes.io/projected/e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446-kube-api-access-92vtl\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tt4nt\" (UID: \"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:37.160865 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:37.160793 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:37.524157 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:37.524131 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt"] Nov 03 22:16:37.526767 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:16:37.526741 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54d8a0d_80b3_40a4_bc7a_cc3ad6e7a446.slice/crio-f845cddfafb8cf64b27a506dc5f7398748b0ec388f39165698544db9a24f0916 WatchSource:0}: Error finding container f845cddfafb8cf64b27a506dc5f7398748b0ec388f39165698544db9a24f0916: Status 404 returned error can't find the container with id f845cddfafb8cf64b27a506dc5f7398748b0ec388f39165698544db9a24f0916 Nov 03 22:16:37.849418 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:37.849333 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" event={"ID":"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446","Type":"ContainerStarted","Data":"f845cddfafb8cf64b27a506dc5f7398748b0ec388f39165698544db9a24f0916"} Nov 03 22:16:38.855056 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:38.855020 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" event={"ID":"e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446","Type":"ContainerStarted","Data":"be7e2aa1c6861ad7c6ab5cd3ffa9584f75421cdb1d2c63c0beb18edbeec3fcc0"} Nov 03 22:16:38.962845 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:38.962790 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" podStartSLOduration=2.332800823 podStartE2EDuration="2.962777805s" totalImagesPullingTime="629.976982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:38.962157286 +0000 UTC m=+518.374343302" watchObservedRunningTime="2026-11-03 22:16:38.962777805 +0000 UTC m=+518.374963822" Nov 03 22:16:39.043423 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.043382 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt"] Nov 03 22:16:39.046903 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.046882 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.078767 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.078740 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhbb\" (UniqueName: \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-kube-api-access-shhbb\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.078873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.078789 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.078873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.078809 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.078966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.078944 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.079002 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.078978 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.079035 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.079004 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.079093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.079081 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.079125 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.079102 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.079160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.079127 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.095679 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.095659 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:16:39.095745 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.095731 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kf4rv\"" Nov 03 22:16:39.101812 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.101790 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt"] Nov 03 22:16:39.161119 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.161045 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:39.180595 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.180571 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhbb\" (UniqueName: \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-kube-api-access-shhbb\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.180710 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.180613 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.180710 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.180629 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.180710 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.180671 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.180710 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.180695 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.180897 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.180714 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.180897 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.180730 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.180897 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.180755 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.180897 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.180785 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.181173 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.181142 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.181292 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.181236 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.181356 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.181301 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.181417 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.181356 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.181607 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.181586 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.182898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.182879 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.183121 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.183101 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.233405 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.233382 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.233721 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.233700 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhbb\" (UniqueName: \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-kube-api-access-shhbb\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-rjhlt\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.359594 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.359565 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:39.770296 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.770272 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt"] Nov 03 22:16:39.772723 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:16:39.772683 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526cc03d_8f45_42a9_b063_f7884ffbf8fe.slice/crio-87bb2f282574ce3050ff8900cf553d297d0aef492d914763b21bd4a85deeb29c WatchSource:0}: Error finding container 87bb2f282574ce3050ff8900cf553d297d0aef492d914763b21bd4a85deeb29c: Status 404 returned error can't find the container with id 87bb2f282574ce3050ff8900cf553d297d0aef492d914763b21bd4a85deeb29c Nov 03 22:16:39.860458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:39.860379 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" event={"ID":"526cc03d-8f45-42a9-b063-f7884ffbf8fe","Type":"ContainerStarted","Data":"87bb2f282574ce3050ff8900cf553d297d0aef492d914763b21bd4a85deeb29c"} Nov 03 22:16:40.538333 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:40.538298 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:40.550479 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:40.550450 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:40.876998 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:40.876925 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:16:40.968770 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:40.968735 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:16:42.338788 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.338754 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n"] Nov 03 22:16:42.342086 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.342048 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.413110 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.413074 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.413273 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.413135 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.413328 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.413278 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.413378 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.413331 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2npz\" (UniqueName: \"kubernetes.io/projected/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-kube-api-access-z2npz\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.413544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.413524 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.413633 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.413619 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.420692 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.420665 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n"] Nov 03 22:16:42.440955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.440930 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 22:16:42.514913 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.514878 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.515113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.514935 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.515113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.514956 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.515113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.514977 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.515113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.515003 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.515113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.515033 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2npz\" (UniqueName: \"kubernetes.io/projected/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-kube-api-access-z2npz\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.515403 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.515375 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.515440 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.515375 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.515664 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.515640 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.517755 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.517733 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.518087 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.518056 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.569044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.569015 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2npz\" (UniqueName: \"kubernetes.io/projected/435e9ee4-e39e-42f5-8a4b-e949f30b71a3-kube-api-access-z2npz\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dql2n\" (UID: \"435e9ee4-e39e-42f5-8a4b-e949f30b71a3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:42.658298 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:42.658223 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:43.027939 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:43.027850 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n"] Nov 03 22:16:43.030537 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:16:43.030489 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435e9ee4_e39e_42f5_8a4b_e949f30b71a3.slice/crio-9d20887e1f565eedc7e5f70dd57774624528490b249871a35a7e25dce7eea607 WatchSource:0}: Error finding container 9d20887e1f565eedc7e5f70dd57774624528490b249871a35a7e25dce7eea607: Status 404 returned error can't find the container with id 9d20887e1f565eedc7e5f70dd57774624528490b249871a35a7e25dce7eea607 Nov 03 22:16:43.881911 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:43.881878 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" event={"ID":"435e9ee4-e39e-42f5-8a4b-e949f30b71a3","Type":"ContainerStarted","Data":"fa3f054ae5d52e35659d065ad9bbfd2f55e67312b9a1d987c6432edb7c60b067"} Nov 03 22:16:43.882320 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:43.881927 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" event={"ID":"435e9ee4-e39e-42f5-8a4b-e949f30b71a3","Type":"ContainerStarted","Data":"9d20887e1f565eedc7e5f70dd57774624528490b249871a35a7e25dce7eea607"} Nov 03 22:16:43.885412 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:43.885382 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:16:43.885485 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:43.885451 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:16:43.885544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:43.885482 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:16:43.992758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:43.992696 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" podStartSLOduration=1.2713905620000001 podStartE2EDuration="1.992682673s" totalImagesPullingTime="721.292111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:43.991708445 +0000 UTC m=+523.403894476" watchObservedRunningTime="2026-11-03 22:16:43.992682673 +0000 UTC m=+523.404868692" Nov 03 22:16:44.663529 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:44.662591 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:44.888027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:44.887990 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" event={"ID":"526cc03d-8f45-42a9-b063-f7884ffbf8fe","Type":"ContainerStarted","Data":"3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb"} Nov 03 22:16:45.059784 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:45.059721 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" podStartSLOduration=1.9500083369999999 podStartE2EDuration="6.059700985s" totalImagesPullingTime="4.109692648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:45.058986257 +0000 UTC m=+524.471172276" watchObservedRunningTime="2026-11-03 22:16:45.059700985 +0000 UTC m=+524.471887002" Nov 03 22:16:45.363909 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:45.363822 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:45.365238 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:45.365203 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:16:45.365478 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:45.365448 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 22:16:46.363873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:46.363835 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:16:46.364259 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:46.363896 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 22:16:47.163707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:47.163666 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:47.176165 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:47.176140 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:47.360927 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:47.360888 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:16:47.361237 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:47.361208 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 22:16:47.912513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:47.912465 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tt4nt" Nov 03 22:16:48.363522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:48.362993 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:16:48.363522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:48.363074 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 22:16:49.363942 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.363899 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:16:49.364460 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.363955 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 22:16:49.675548 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.675469 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf"] Nov 03 22:16:49.678686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.678669 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.727724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.727694 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 22:16:49.775684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.775647 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf"] Nov 03 22:16:49.808428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.808397 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.808649 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.808469 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gpr7\" (UniqueName: \"kubernetes.io/projected/608ba6ab-1275-43bf-9f86-ac190f8b8a99-kube-api-access-8gpr7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.808723 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.808675 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.808782 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.808763 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.808841 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.808828 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.808904 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.808876 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/608ba6ab-1275-43bf-9f86-ac190f8b8a99-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.910110 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.910072 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gpr7\" (UniqueName: \"kubernetes.io/projected/608ba6ab-1275-43bf-9f86-ac190f8b8a99-kube-api-access-8gpr7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.910110 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.910116 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.910380 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.910150 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.910380 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.910186 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.910380 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.910208 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/608ba6ab-1275-43bf-9f86-ac190f8b8a99-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.910380 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.910256 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.910728 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.910698 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.910811 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.910754 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.910895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.910875 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.912682 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.912660 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/608ba6ab-1275-43bf-9f86-ac190f8b8a99-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.913143 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.913114 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/608ba6ab-1275-43bf-9f86-ac190f8b8a99-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.964872 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.964849 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gpr7\" (UniqueName: \"kubernetes.io/projected/608ba6ab-1275-43bf-9f86-ac190f8b8a99-kube-api-access-8gpr7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf\" (UID: \"608ba6ab-1275-43bf-9f86-ac190f8b8a99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:49.990928 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:49.990901 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:50.356188 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:50.356119 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf"] Nov 03 22:16:50.359589 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:16:50.359550 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608ba6ab_1275_43bf_9f86_ac190f8b8a99.slice/crio-2ff63baa1041a308c8fab992737935d5f7b5e0eb7481746ccf1afa845281914a WatchSource:0}: Error finding container 2ff63baa1041a308c8fab992737935d5f7b5e0eb7481746ccf1afa845281914a: Status 404 returned error can't find the container with id 2ff63baa1041a308c8fab992737935d5f7b5e0eb7481746ccf1afa845281914a Nov 03 22:16:50.359848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:50.359826 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:16:50.359926 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:50.359876 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 22:16:50.918167 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:50.918129 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" event={"ID":"608ba6ab-1275-43bf-9f86-ac190f8b8a99","Type":"ContainerStarted","Data":"2ff63baa1041a308c8fab992737935d5f7b5e0eb7481746ccf1afa845281914a"} Nov 03 22:16:51.866643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:51.866608 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:51.867017 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:51.866990 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:51.867975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:51.867937 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:16:51.924029 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:51.923991 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" event={"ID":"608ba6ab-1275-43bf-9f86-ac190f8b8a99","Type":"ContainerStarted","Data":"f5097e351f7022f97270726fff145086c8a26d7d416dc7349985dfac5e62c990"} Nov 03 22:16:51.991927 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:51.991900 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:16:52.237861 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:52.237809 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" podStartSLOduration=2.615407575 podStartE2EDuration="3.237794285s" totalImagesPullingTime="622.38671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:52.236580775 +0000 UTC m=+531.648766792" watchObservedRunningTime="2026-11-03 22:16:52.237794285 +0000 UTC m=+531.649980302" Nov 03 22:16:52.658565 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:52.658454 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:52.671311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:52.671283 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:52.814709 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:52.814675 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:16:52.939914 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:52.939886 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dql2n" Nov 03 22:16:56.500387 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.500352 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l"] Nov 03 22:16:56.504099 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.504080 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.577530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.577487 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.577687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.577588 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac4a706f-973b-40e0-a54f-56da8b520e96-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.577687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.577650 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.577687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.577675 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.577866 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.577839 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbvd\" (UniqueName: \"kubernetes.io/projected/ac4a706f-973b-40e0-a54f-56da8b520e96-kube-api-access-lmbvd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.577957 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.577937 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.588004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.587983 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 22:16:56.603593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.603565 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l"] Nov 03 22:16:56.679342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.679313 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.679513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.679351 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.679513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.679446 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbvd\" (UniqueName: \"kubernetes.io/projected/ac4a706f-973b-40e0-a54f-56da8b520e96-kube-api-access-lmbvd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.679622 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.679519 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.679622 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.679545 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.679622 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.679583 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac4a706f-973b-40e0-a54f-56da8b520e96-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.679805 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.679780 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.679965 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.679946 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.680033 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.679863 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.681584 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.681564 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac4a706f-973b-40e0-a54f-56da8b520e96-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.681920 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.681902 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac4a706f-973b-40e0-a54f-56da8b520e96-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.732690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.732662 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbvd\" (UniqueName: \"kubernetes.io/projected/ac4a706f-973b-40e0-a54f-56da8b520e96-kube-api-access-lmbvd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l\" (UID: \"ac4a706f-973b-40e0-a54f-56da8b520e96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:56.818466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:56.818369 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:57.185555 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:57.185531 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l"] Nov 03 22:16:57.190908 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:16:57.190873 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4a706f_973b_40e0_a54f_56da8b520e96.slice/crio-36a7da6aaa6a7fe667f7f89a9d45aea7dae0d8601d647a005217d3fc8a750d9b WatchSource:0}: Error finding container 36a7da6aaa6a7fe667f7f89a9d45aea7dae0d8601d647a005217d3fc8a750d9b: Status 404 returned error can't find the container with id 36a7da6aaa6a7fe667f7f89a9d45aea7dae0d8601d647a005217d3fc8a750d9b Nov 03 22:16:57.952985 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:57.952953 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" event={"ID":"ac4a706f-973b-40e0-a54f-56da8b520e96","Type":"ContainerStarted","Data":"dd64e247ad5a24e132488281f1ba39fc83cd1bf5f54d3bee2352d96671d5d34a"} Nov 03 22:16:57.953432 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:57.953004 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" event={"ID":"ac4a706f-973b-40e0-a54f-56da8b520e96","Type":"ContainerStarted","Data":"36a7da6aaa6a7fe667f7f89a9d45aea7dae0d8601d647a005217d3fc8a750d9b"} Nov 03 22:16:58.062391 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:58.062294 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" podStartSLOduration=1.445476256 podStartE2EDuration="2.062280696s" totalImagesPullingTime="616.80444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:58.061357231 +0000 UTC m=+537.473543248" watchObservedRunningTime="2026-11-03 22:16:58.062280696 +0000 UTC m=+537.474466712" Nov 03 22:16:58.818491 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:58.818460 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:16:58.994402 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:16:58.994363 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:16:59.991266 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:16:59.991232 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:17:00.003579 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:00.003553 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:17:00.980052 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:00.980019 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf" Nov 03 22:17:06.819281 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:06.819247 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:17:06.831597 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:06.831574 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:17:07.004397 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:07.004364 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l" Nov 03 22:17:08.687838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:08.687802 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:17:09.311392 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.311352 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt"] Nov 03 22:17:09.317379 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.317351 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.404675 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.404639 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.404855 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.404688 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9h4s\" (UniqueName: \"kubernetes.io/projected/fce20530-f3fd-4d1a-9dcd-151248db648a-kube-api-access-v9h4s\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.404855 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.404794 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.404855 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.404834 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.405001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.404905 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce20530-f3fd-4d1a-9dcd-151248db648a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.405001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.404930 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.405001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.404948 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.405104 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.405047 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.405153 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.405134 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.408744 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.408710 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt"] Nov 03 22:17:09.506010 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.505979 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506176 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506014 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9h4s\" (UniqueName: \"kubernetes.io/projected/fce20530-f3fd-4d1a-9dcd-151248db648a-kube-api-access-v9h4s\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506176 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506138 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506295 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506184 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506295 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506208 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce20530-f3fd-4d1a-9dcd-151248db648a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506352 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506371 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506378 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506488 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506555 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506570 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506752 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506657 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.506897 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.506872 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce20530-f3fd-4d1a-9dcd-151248db648a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.507063 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.507041 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.508603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.508579 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.509035 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.509008 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.559309 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.559288 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9h4s\" (UniqueName: \"kubernetes.io/projected/fce20530-f3fd-4d1a-9dcd-151248db648a-kube-api-access-v9h4s\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.559411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.559308 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce20530-f3fd-4d1a-9dcd-151248db648a-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-bqdwt\" (UID: \"fce20530-f3fd-4d1a-9dcd-151248db648a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:09.630160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:09.630076 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:10.087519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:10.087477 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt"] Nov 03 22:17:10.089518 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:17:10.089468 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce20530_f3fd_4d1a_9dcd_151248db648a.slice/crio-7b31bf6c72a2d2e926558003130295bf4fe561e80c32333fc5b9d31c3a1d07fb WatchSource:0}: Error finding container 7b31bf6c72a2d2e926558003130295bf4fe561e80c32333fc5b9d31c3a1d07fb: Status 404 returned error can't find the container with id 7b31bf6c72a2d2e926558003130295bf4fe561e80c32333fc5b9d31c3a1d07fb Nov 03 22:17:10.091750 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:10.091721 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:17:10.091839 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:10.091784 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:17:10.091839 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:10.091810 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:17:11.014072 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:11.014034 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" event={"ID":"fce20530-f3fd-4d1a-9dcd-151248db648a","Type":"ContainerStarted","Data":"cb9c58fac8bb5ed210bd3ddd209672f8eb1b0889094b7128c234853b334974c7"} Nov 03 22:17:11.014278 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:11.014089 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" event={"ID":"fce20530-f3fd-4d1a-9dcd-151248db648a","Type":"ContainerStarted","Data":"7b31bf6c72a2d2e926558003130295bf4fe561e80c32333fc5b9d31c3a1d07fb"} Nov 03 22:17:11.156515 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:11.155066 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" podStartSLOduration=2.155032043 podStartE2EDuration="2.155032043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:11.138579298 +0000 UTC m=+550.550765319" watchObservedRunningTime="2026-11-03 22:17:11.155032043 +0000 UTC m=+550.567218060" Nov 03 22:17:11.631284 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:11.631237 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:11.632768 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:11.632731 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:17:11.632903 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:11.632812 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" podUID="fce20530-f3fd-4d1a-9dcd-151248db648a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 22:17:12.630810 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:12.630775 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:17:12.631245 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:12.630830 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" podUID="fce20530-f3fd-4d1a-9dcd-151248db648a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 22:17:13.631472 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:13.631437 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:17:13.632007 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:13.631546 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" podUID="fce20530-f3fd-4d1a-9dcd-151248db648a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 22:17:14.631310 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:14.631271 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:17:14.631547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:14.631320 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" podUID="fce20530-f3fd-4d1a-9dcd-151248db648a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 22:17:15.630896 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:15.630857 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:17:15.631075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:15.630913 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" podUID="fce20530-f3fd-4d1a-9dcd-151248db648a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 22:17:16.630643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:16.630600 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:17:16.631159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:16.630654 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" podUID="fce20530-f3fd-4d1a-9dcd-151248db648a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 22:17:18.277598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:18.277565 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:18.278088 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:18.277966 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:18.278832 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:18.278806 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" Nov 03 22:17:24.442771 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.442736 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g"] Nov 03 22:17:24.450630 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.450604 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.470452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.470424 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.470583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.470457 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.470583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.470476 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.470583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.470563 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.470752 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.470592 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqh7\" (UniqueName: \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-kube-api-access-pxqh7\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.470752 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.470707 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.470862 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.470753 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.470862 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.470813 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.470944 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.470869 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.572158 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572127 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.572158 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572165 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.572436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572193 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.572436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572223 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.572436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572247 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqh7\" (UniqueName: \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-kube-api-access-pxqh7\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.572436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572272 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.572436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572300 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.572848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572339 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.572908 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572863 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.572969 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572942 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.573024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572975 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.573024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.572680 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.573236 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.573211 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.573321 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.573301 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.574596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.574568 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.575036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.575019 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.625431 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.625397 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.625700 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.625680 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqh7\" (UniqueName: \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-kube-api-access-pxqh7\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-ghb4g\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:24.629823 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.629801 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g"] Nov 03 22:17:24.764248 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:24.764141 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:25.173884 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:25.173860 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g"] Nov 03 22:17:25.176180 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:17:25.176143 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a7dfc0_3a48_49cf_8e05_e1b9799f08ac.slice/crio-34af7a863a515da534e11b22911fd76e05b6b9a995620020e674fe9e458d9d65 WatchSource:0}: Error finding container 34af7a863a515da534e11b22911fd76e05b6b9a995620020e674fe9e458d9d65: Status 404 returned error can't find the container with id 34af7a863a515da534e11b22911fd76e05b6b9a995620020e674fe9e458d9d65 Nov 03 22:17:25.178753 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:25.178722 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:17:25.178844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:25.178792 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:17:25.178844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:25.178818 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:17:26.083788 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:26.083753 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" event={"ID":"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac","Type":"ContainerStarted","Data":"2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714"} Nov 03 22:17:26.084182 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:26.083812 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" event={"ID":"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac","Type":"ContainerStarted","Data":"34af7a863a515da534e11b22911fd76e05b6b9a995620020e674fe9e458d9d65"} Nov 03 22:17:26.291704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:26.291648 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" podStartSLOduration=2.291633919 podStartE2EDuration="2.291633919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:26.243985405 +0000 UTC m=+565.656171434" watchObservedRunningTime="2026-11-03 22:17:26.291633919 +0000 UTC m=+565.703819931" Nov 03 22:17:26.765106 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:26.765071 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:26.766554 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:26.766527 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:17:26.766677 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:26.766574 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 22:17:27.764927 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:27.764893 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:17:27.765438 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:27.764944 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 22:17:27.795414 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:27.795384 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-656fb666db-8fhmz"] Nov 03 22:17:27.798574 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:27.798555 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-656fb666db-8fhmz" Nov 03 22:17:27.865520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:27.859349 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-656fb666db-8fhmz"] Nov 03 22:17:27.906630 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:27.906596 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjv72\" (UniqueName: \"kubernetes.io/projected/ef51a905-8e84-496a-8b7e-02cbdd3a2395-kube-api-access-hjv72\") pod \"authorino-656fb666db-8fhmz\" (UID: \"ef51a905-8e84-496a-8b7e-02cbdd3a2395\") " pod="kuadrant-system/authorino-656fb666db-8fhmz" Nov 03 22:17:27.906963 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:27.906697 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef51a905-8e84-496a-8b7e-02cbdd3a2395-tls-cert\") pod \"authorino-656fb666db-8fhmz\" (UID: \"ef51a905-8e84-496a-8b7e-02cbdd3a2395\") " pod="kuadrant-system/authorino-656fb666db-8fhmz" Nov 03 22:17:28.007804 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:28.007769 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjv72\" (UniqueName: \"kubernetes.io/projected/ef51a905-8e84-496a-8b7e-02cbdd3a2395-kube-api-access-hjv72\") pod \"authorino-656fb666db-8fhmz\" (UID: \"ef51a905-8e84-496a-8b7e-02cbdd3a2395\") " pod="kuadrant-system/authorino-656fb666db-8fhmz" Nov 03 22:17:28.007984 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:28.007818 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef51a905-8e84-496a-8b7e-02cbdd3a2395-tls-cert\") pod \"authorino-656fb666db-8fhmz\" (UID: \"ef51a905-8e84-496a-8b7e-02cbdd3a2395\") " pod="kuadrant-system/authorino-656fb666db-8fhmz" Nov 03 22:17:28.010535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:28.010487 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef51a905-8e84-496a-8b7e-02cbdd3a2395-tls-cert\") pod \"authorino-656fb666db-8fhmz\" (UID: \"ef51a905-8e84-496a-8b7e-02cbdd3a2395\") " pod="kuadrant-system/authorino-656fb666db-8fhmz" Nov 03 22:17:28.063583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:28.063480 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjv72\" (UniqueName: \"kubernetes.io/projected/ef51a905-8e84-496a-8b7e-02cbdd3a2395-kube-api-access-hjv72\") pod \"authorino-656fb666db-8fhmz\" (UID: \"ef51a905-8e84-496a-8b7e-02cbdd3a2395\") " pod="kuadrant-system/authorino-656fb666db-8fhmz" Nov 03 22:17:28.110338 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:28.110307 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-656fb666db-8fhmz" Nov 03 22:17:28.496684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:28.496649 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-656fb666db-8fhmz"] Nov 03 22:17:28.499871 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:17:28.499842 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef51a905_8e84_496a_8b7e_02cbdd3a2395.slice/crio-7c1b6b785ec3ae1016587a5bda74082bb02f5dbd48b61440804017d749ec1642 WatchSource:0}: Error finding container 7c1b6b785ec3ae1016587a5bda74082bb02f5dbd48b61440804017d749ec1642: Status 404 returned error can't find the container with id 7c1b6b785ec3ae1016587a5bda74082bb02f5dbd48b61440804017d749ec1642 Nov 03 22:17:28.764907 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:28.764830 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:17:28.764907 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:28.764882 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 22:17:28.996129 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:17:28.996090 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:17:29.098473 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:29.098385 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-656fb666db-8fhmz" event={"ID":"ef51a905-8e84-496a-8b7e-02cbdd3a2395","Type":"ContainerStarted","Data":"7c1b6b785ec3ae1016587a5bda74082bb02f5dbd48b61440804017d749ec1642"} Nov 03 22:17:29.765041 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:29.765011 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:17:29.765214 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:29.765061 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 22:17:30.764918 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:30.764878 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:17:30.765114 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:30.764943 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 22:17:31.111109 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.111012 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-656fb666db-8fhmz" event={"ID":"ef51a905-8e84-496a-8b7e-02cbdd3a2395","Type":"ContainerStarted","Data":"2cdf4dc1542aa6adf33c099507dfa3daa300c19e9d76564e5b0e945ebe99ada4"} Nov 03 22:17:31.217212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.217156 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-656fb666db-8fhmz" podStartSLOduration=2.3002235 podStartE2EDuration="4.217137862s" totalImagesPullingTime="1.916914362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:31.216124681 +0000 UTC m=+570.628310699" watchObservedRunningTime="2026-11-03 22:17:31.217137862 +0000 UTC m=+570.629323883" Nov 03 22:17:31.265830 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.265792 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7478998589-vcrpp"] Nov 03 22:17:31.266067 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.266031 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7478998589-vcrpp" podUID="ab80cc47-9e3e-4d3b-9220-83134c8f38a9" containerName="authorino" containerID="cri-o://36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87" gracePeriod=30 Nov 03 22:17:31.535987 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.535950 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7478998589-vcrpp" Nov 03 22:17:31.650973 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.650893 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-tls-cert\" (UniqueName: \"kubernetes.io/secret/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-tls-cert\") pod \"ab80cc47-9e3e-4d3b-9220-83134c8f38a9\" (UID: \"ab80cc47-9e3e-4d3b-9220-83134c8f38a9\") " Nov 03 22:17:31.651147 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.650991 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-kube-api-access-4468t\" (UniqueName: \"kubernetes.io/projected/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-kube-api-access-4468t\") pod \"ab80cc47-9e3e-4d3b-9220-83134c8f38a9\" (UID: \"ab80cc47-9e3e-4d3b-9220-83134c8f38a9\") " Nov 03 22:17:31.653460 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.653422 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-kube-api-access-4468t" pod "ab80cc47-9e3e-4d3b-9220-83134c8f38a9" (UID: "ab80cc47-9e3e-4d3b-9220-83134c8f38a9"). InnerVolumeSpecName "kube-api-access-4468t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:17:31.662434 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.662410 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-tls-cert" pod "ab80cc47-9e3e-4d3b-9220-83134c8f38a9" (UID: "ab80cc47-9e3e-4d3b-9220-83134c8f38a9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:17:31.752038 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.752001 6234 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-tls-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:17:31.752038 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.752032 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4468t\" (UniqueName: \"kubernetes.io/projected/ab80cc47-9e3e-4d3b-9220-83134c8f38a9-kube-api-access-4468t\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:17:31.765030 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.764993 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:17:31.765189 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:31.765063 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 22:17:32.118977 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:32.118938 6234 generic.go:356] "Generic (PLEG): container finished" podID="ab80cc47-9e3e-4d3b-9220-83134c8f38a9" containerID="36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87" exitCode=0 Nov 03 22:17:32.119469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:32.119010 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7478998589-vcrpp" event={"ID":"ab80cc47-9e3e-4d3b-9220-83134c8f38a9","Type":"ContainerDied","Data":"36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87"} Nov 03 22:17:32.119469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:32.119022 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7478998589-vcrpp" Nov 03 22:17:32.119469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:32.119054 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7478998589-vcrpp" event={"ID":"ab80cc47-9e3e-4d3b-9220-83134c8f38a9","Type":"ContainerDied","Data":"5acd808922c41ce70e65f482802978cce5558d33f5b892d4d981c8935e91bd04"} Nov 03 22:17:32.119469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:32.119077 6234 scope.go:122] "RemoveContainer" containerID="36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87" Nov 03 22:17:32.130884 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:32.130856 6234 scope.go:122] "RemoveContainer" containerID="36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87" Nov 03 22:17:32.131266 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:17:32.131243 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87\": container with ID starting with 36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87 not found: ID does not exist" containerID="36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87" Nov 03 22:17:32.131349 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:32.131275 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87"} err="failed to get container status \"36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87\": rpc error: code = NotFound desc = could not find container \"36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87\": container with ID starting with 36c5139d987a9f0bf2b2e956156064540653e32e4cc0500e8ca64b4af5dbcd87 not found: ID does not exist" Nov 03 22:17:32.271643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:32.271609 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7478998589-vcrpp"] Nov 03 22:17:32.273326 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:32.273294 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7478998589-vcrpp"] Nov 03 22:17:32.702976 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:32.702928 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab80cc47-9e3e-4d3b-9220-83134c8f38a9" path="/var/lib/kubelet/pods/ab80cc47-9e3e-4d3b-9220-83134c8f38a9/volumes" Nov 03 22:17:33.580519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:33.580468 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:33.581631 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:33.581612 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:33.582693 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:17:33.582672 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:17:58.995320 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:17:58.995282 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:18:28.995675 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:18:28.995638 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:18:40.253536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.253487 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk"] Nov 03 22:18:40.253966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.253925 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ab80cc47-9e3e-4d3b-9220-83134c8f38a9" containerName="authorino" Nov 03 22:18:40.253966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.253936 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab80cc47-9e3e-4d3b-9220-83134c8f38a9" containerName="authorino" Nov 03 22:18:40.254036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.254009 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab80cc47-9e3e-4d3b-9220-83134c8f38a9" containerName="authorino" Nov 03 22:18:40.257293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.257271 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:18:40.307700 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.307663 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-f81f8f\"" Nov 03 22:18:40.354524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.354446 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-f81f8f-dockercfg-stm5n\"" Nov 03 22:18:40.356656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.356617 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk"] Nov 03 22:18:40.425711 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.425679 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9864w\" (UniqueName: \"kubernetes.io/projected/6d154f05-576c-44a6-9301-9707cc2f629a-kube-api-access-9864w\") pod \"payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk\" (UID: \"6d154f05-576c-44a6-9301-9707cc2f629a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:18:40.425905 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.425719 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d154f05-576c-44a6-9301-9707cc2f629a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk\" (UID: \"6d154f05-576c-44a6-9301-9707cc2f629a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:18:40.428969 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.428941 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj"] Nov 03 22:18:40.432565 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.432548 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:18:40.502854 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.502822 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj"] Nov 03 22:18:40.527156 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.527090 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9864w\" (UniqueName: \"kubernetes.io/projected/6d154f05-576c-44a6-9301-9707cc2f629a-kube-api-access-9864w\") pod \"payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk\" (UID: \"6d154f05-576c-44a6-9301-9707cc2f629a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:18:40.527156 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.527127 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d154f05-576c-44a6-9301-9707cc2f629a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk\" (UID: \"6d154f05-576c-44a6-9301-9707cc2f629a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:18:40.527853 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.527833 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d154f05-576c-44a6-9301-9707cc2f629a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk\" (UID: \"6d154f05-576c-44a6-9301-9707cc2f629a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:18:40.581047 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.581016 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9864w\" (UniqueName: \"kubernetes.io/projected/6d154f05-576c-44a6-9301-9707cc2f629a-kube-api-access-9864w\") pod \"payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk\" (UID: \"6d154f05-576c-44a6-9301-9707cc2f629a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:18:40.627979 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.627950 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mttv7\" (UniqueName: \"kubernetes.io/projected/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-kube-api-access-mttv7\") pod \"payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj\" (UID: \"9ce525c2-9b52-4243-9b67-1d34d58aa1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:18:40.628105 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.627991 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj\" (UID: \"9ce525c2-9b52-4243-9b67-1d34d58aa1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:18:40.728829 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.728795 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mttv7\" (UniqueName: \"kubernetes.io/projected/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-kube-api-access-mttv7\") pod \"payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj\" (UID: \"9ce525c2-9b52-4243-9b67-1d34d58aa1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:18:40.729009 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.728840 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj\" (UID: \"9ce525c2-9b52-4243-9b67-1d34d58aa1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:18:40.729544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.729522 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj\" (UID: \"9ce525c2-9b52-4243-9b67-1d34d58aa1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:18:40.781869 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.781810 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mttv7\" (UniqueName: \"kubernetes.io/projected/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-kube-api-access-mttv7\") pod \"payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj\" (UID: \"9ce525c2-9b52-4243-9b67-1d34d58aa1c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:18:40.869469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:40.869441 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:18:41.045706 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.045626 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:18:41.156936 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.156902 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp"] Nov 03 22:18:41.161388 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.161368 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.212632 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.212582 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-3378fd-serving-cert\"" Nov 03 22:18:41.214702 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.214675 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp"] Nov 03 22:18:41.261993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.261971 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk"] Nov 03 22:18:41.264694 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:18:41.264665 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d154f05_576c_44a6_9301_9707cc2f629a.slice/crio-4044bf8a04291595a12e2928caf158446532466cea430d63f962e6bcd264a401 WatchSource:0}: Error finding container 4044bf8a04291595a12e2928caf158446532466cea430d63f962e6bcd264a401: Status 404 returned error can't find the container with id 4044bf8a04291595a12e2928caf158446532466cea430d63f962e6bcd264a401 Nov 03 22:18:41.335299 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.335210 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-metrics-certs\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.335299 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.335266 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5163c5b4-4805-4adc-9216-8958b17a2b6d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.335485 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.335374 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6s8\" (UniqueName: \"kubernetes.io/projected/5163c5b4-4805-4adc-9216-8958b17a2b6d-kube-api-access-dw6s8\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.335543 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.335481 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.415962 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.415932 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj"] Nov 03 22:18:41.418787 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:18:41.418753 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce525c2_9b52_4243_9b67_1d34d58aa1c9.slice/crio-ef6ff40052594a899f7eb52e0ebbdc158f9fa7ad4c76d4ccc9a0a2bff879d898 WatchSource:0}: Error finding container ef6ff40052594a899f7eb52e0ebbdc158f9fa7ad4c76d4ccc9a0a2bff879d898: Status 404 returned error can't find the container with id ef6ff40052594a899f7eb52e0ebbdc158f9fa7ad4c76d4ccc9a0a2bff879d898 Nov 03 22:18:41.436098 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.436073 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6s8\" (UniqueName: \"kubernetes.io/projected/5163c5b4-4805-4adc-9216-8958b17a2b6d-kube-api-access-dw6s8\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.436236 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.436221 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.436292 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.436276 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-metrics-certs\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.436398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.436313 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5163c5b4-4805-4adc-9216-8958b17a2b6d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.437110 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.437085 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5163c5b4-4805-4adc-9216-8958b17a2b6d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.438725 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.438702 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.438826 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.438728 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-metrics-certs\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.451568 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.451525 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" event={"ID":"9ce525c2-9b52-4243-9b67-1d34d58aa1c9","Type":"ContainerStarted","Data":"ef6ff40052594a899f7eb52e0ebbdc158f9fa7ad4c76d4ccc9a0a2bff879d898"} Nov 03 22:18:41.452682 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.452660 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" event={"ID":"6d154f05-576c-44a6-9301-9707cc2f629a","Type":"ContainerStarted","Data":"4044bf8a04291595a12e2928caf158446532466cea430d63f962e6bcd264a401"} Nov 03 22:18:41.489909 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.489884 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6s8\" (UniqueName: \"kubernetes.io/projected/5163c5b4-4805-4adc-9216-8958b17a2b6d-kube-api-access-dw6s8\") pod \"maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:41.774733 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:41.774698 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:42.139396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:42.139368 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp"] Nov 03 22:18:42.142247 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:18:42.142212 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5163c5b4_4805_4adc_9216_8958b17a2b6d.slice/crio-c14c973a42aff988d8df9feb227a4479a3101ebac53c1987834a14cf76479025 WatchSource:0}: Error finding container c14c973a42aff988d8df9feb227a4479a3101ebac53c1987834a14cf76479025: Status 404 returned error can't find the container with id c14c973a42aff988d8df9feb227a4479a3101ebac53c1987834a14cf76479025 Nov 03 22:18:42.459592 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:42.459533 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" event={"ID":"5163c5b4-4805-4adc-9216-8958b17a2b6d","Type":"ContainerStarted","Data":"c14c973a42aff988d8df9feb227a4479a3101ebac53c1987834a14cf76479025"} Nov 03 22:18:43.470819 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:43.470779 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" event={"ID":"5163c5b4-4805-4adc-9216-8958b17a2b6d","Type":"ContainerStarted","Data":"e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d"} Nov 03 22:18:43.471239 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:43.470955 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:44.478750 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:44.478719 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" event={"ID":"9ce525c2-9b52-4243-9b67-1d34d58aa1c9","Type":"ContainerStarted","Data":"ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915"} Nov 03 22:18:44.479083 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:44.478937 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:18:44.585527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:44.585464 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" podStartSLOduration=1.6265236920000001 podStartE2EDuration="4.585451468s" totalImagesPullingTime="2.958927776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:44.584979213 +0000 UTC m=+643.997165230" watchObservedRunningTime="2026-11-03 22:18:44.585451468 +0000 UTC m=+643.997637527" Nov 03 22:18:44.586430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:44.586400 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" podStartSLOduration=2.564792671 podStartE2EDuration="3.586392432s" totalImagesPullingTime="1.021599761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:43.580377877 +0000 UTC m=+642.992563897" watchObservedRunningTime="2026-11-03 22:18:44.586392432 +0000 UTC m=+643.998578452" Nov 03 22:18:45.378043 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.378015 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss"] Nov 03 22:18:45.381740 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.381713 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:18:45.433787 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.433758 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-3378fd\"" Nov 03 22:18:45.434897 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.434873 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss"] Nov 03 22:18:45.479625 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.479590 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-3378fd-dockercfg-2m8ql\"" Nov 03 22:18:45.480173 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.480040 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea59b41-386c-489e-811c-01f107d9458f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss\" (UID: \"3ea59b41-386c-489e-811c-01f107d9458f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:18:45.480431 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.480260 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8gr\" (UniqueName: \"kubernetes.io/projected/3ea59b41-386c-489e-811c-01f107d9458f-kube-api-access-sj8gr\") pod \"payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss\" (UID: \"3ea59b41-386c-489e-811c-01f107d9458f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:18:45.487445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.487412 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" event={"ID":"6d154f05-576c-44a6-9301-9707cc2f629a","Type":"ContainerStarted","Data":"20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9"} Nov 03 22:18:45.525615 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.525583 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g"] Nov 03 22:18:45.529369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.529345 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:18:45.581129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.581102 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g\" (UID: \"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:18:45.581257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.581168 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea59b41-386c-489e-811c-01f107d9458f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss\" (UID: \"3ea59b41-386c-489e-811c-01f107d9458f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:18:45.581257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.581223 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvh2t\" (UniqueName: \"kubernetes.io/projected/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-kube-api-access-rvh2t\") pod \"payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g\" (UID: \"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:18:45.582097 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.581249 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8gr\" (UniqueName: \"kubernetes.io/projected/3ea59b41-386c-489e-811c-01f107d9458f-kube-api-access-sj8gr\") pod \"payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss\" (UID: \"3ea59b41-386c-489e-811c-01f107d9458f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:18:45.582097 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.582019 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea59b41-386c-489e-811c-01f107d9458f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss\" (UID: \"3ea59b41-386c-489e-811c-01f107d9458f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:18:45.584174 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.584149 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g"] Nov 03 22:18:45.599467 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.599424 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" podStartSLOduration=2.411780136 podStartE2EDuration="5.599409531s" totalImagesPullingTime="3.187629395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:45.597450172 +0000 UTC m=+645.009636190" watchObservedRunningTime="2026-11-03 22:18:45.599409531 +0000 UTC m=+645.011595547" Nov 03 22:18:45.634897 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.634834 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8gr\" (UniqueName: \"kubernetes.io/projected/3ea59b41-386c-489e-811c-01f107d9458f-kube-api-access-sj8gr\") pod \"payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss\" (UID: \"3ea59b41-386c-489e-811c-01f107d9458f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:18:45.682379 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.682351 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvh2t\" (UniqueName: \"kubernetes.io/projected/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-kube-api-access-rvh2t\") pod \"payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g\" (UID: \"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:18:45.682489 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.682454 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g\" (UID: \"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:18:45.683119 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.683101 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g\" (UID: \"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:18:45.694259 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.694235 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:18:45.736485 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.736453 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvh2t\" (UniqueName: \"kubernetes.io/projected/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-kube-api-access-rvh2t\") pod \"payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g\" (UID: \"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:18:45.842386 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:45.842350 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:18:46.062288 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.062262 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss"] Nov 03 22:18:46.064957 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:18:46.064930 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea59b41_386c_489e_811c_01f107d9458f.slice/crio-d873029a57d6cb94fbc5a2ce0364035b699e515385f78feae13613c18944e3ad WatchSource:0}: Error finding container d873029a57d6cb94fbc5a2ce0364035b699e515385f78feae13613c18944e3ad: Status 404 returned error can't find the container with id d873029a57d6cb94fbc5a2ce0364035b699e515385f78feae13613c18944e3ad Nov 03 22:18:46.205024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.205003 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g"] Nov 03 22:18:46.207518 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:18:46.207474 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc0ea6a_242f_44a2_aa04_e6b2c3b22735.slice/crio-a0d203bd220fc0c1018e408b8ec1a6233ef9ad68d38065ddf95c01ea77ae0ecf WatchSource:0}: Error finding container a0d203bd220fc0c1018e408b8ec1a6233ef9ad68d38065ddf95c01ea77ae0ecf: Status 404 returned error can't find the container with id a0d203bd220fc0c1018e408b8ec1a6233ef9ad68d38065ddf95c01ea77ae0ecf Nov 03 22:18:46.246294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.246269 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8"] Nov 03 22:18:46.251320 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.251300 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.286905 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.286876 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78wgf\" (UniqueName: \"kubernetes.io/projected/7d7a8dea-f57e-40f0-8dfc-751843570fde-kube-api-access-78wgf\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.287022 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.286932 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-maas-api-tls\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.287185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.287127 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a8dea-f57e-40f0-8dfc-751843570fde-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.287265 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.287201 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-metrics-certs\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.305540 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.301672 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-906673-serving-cert\"" Nov 03 22:18:46.305540 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.305363 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8"] Nov 03 22:18:46.388225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.388189 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-maas-api-tls\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.388416 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.388298 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a8dea-f57e-40f0-8dfc-751843570fde-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.388416 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:18:46.388332 6234 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-serving-cert: secret "maas-api-e2e-worker-w2-906673-serving-cert" not found Nov 03 22:18:46.388416 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:18:46.388408 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-maas-api-tls podName:7d7a8dea-f57e-40f0-8dfc-751843570fde nodeName:}" failed. No retries permitted until 2026-11-03 22:18:46.88838504 +0000 UTC m=+646.300571058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-maas-api-tls") pod "maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" (UID: "7d7a8dea-f57e-40f0-8dfc-751843570fde") : secret "maas-api-e2e-worker-w2-906673-serving-cert" not found Nov 03 22:18:46.388588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.388336 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-metrics-certs\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.388629 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.388593 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78wgf\" (UniqueName: \"kubernetes.io/projected/7d7a8dea-f57e-40f0-8dfc-751843570fde-kube-api-access-78wgf\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.389014 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.388993 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a8dea-f57e-40f0-8dfc-751843570fde-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.390996 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.390972 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-metrics-certs\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.445350 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.445322 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78wgf\" (UniqueName: \"kubernetes.io/projected/7d7a8dea-f57e-40f0-8dfc-751843570fde-kube-api-access-78wgf\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.496316 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.496214 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" event={"ID":"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735","Type":"ContainerStarted","Data":"bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d"} Nov 03 22:18:46.496855 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.496826 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" event={"ID":"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735","Type":"ContainerStarted","Data":"a0d203bd220fc0c1018e408b8ec1a6233ef9ad68d38065ddf95c01ea77ae0ecf"} Nov 03 22:18:46.496976 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.496871 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:18:46.498325 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.498298 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" event={"ID":"3ea59b41-386c-489e-811c-01f107d9458f","Type":"ContainerStarted","Data":"6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db"} Nov 03 22:18:46.498423 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.498352 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" event={"ID":"3ea59b41-386c-489e-811c-01f107d9458f","Type":"ContainerStarted","Data":"d873029a57d6cb94fbc5a2ce0364035b699e515385f78feae13613c18944e3ad"} Nov 03 22:18:46.498423 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.498395 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:18:46.498656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.498639 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:18:46.603690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.603635 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" podStartSLOduration=1.603622057 podStartE2EDuration="1.603622057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:46.602312871 +0000 UTC m=+646.014498889" watchObservedRunningTime="2026-11-03 22:18:46.603622057 +0000 UTC m=+646.015808088" Nov 03 22:18:46.708702 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.708654 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" podStartSLOduration=1.708641589 podStartE2EDuration="1.708641589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:46.708442326 +0000 UTC m=+646.120628344" watchObservedRunningTime="2026-11-03 22:18:46.708641589 +0000 UTC m=+646.120827605" Nov 03 22:18:46.895728 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.895644 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-maas-api-tls\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:46.898018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:46.897989 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-maas-api-tls\") pod \"maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:47.175910 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:47.175835 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:47.540622 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:47.540598 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8"] Nov 03 22:18:47.543224 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:18:47.543191 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7a8dea_f57e_40f0_8dfc_751843570fde.slice/crio-1710dc6abccc4af38e6d82e9e5d7be938ac95f7e9781c1cc4532b35b9cbe8ccb WatchSource:0}: Error finding container 1710dc6abccc4af38e6d82e9e5d7be938ac95f7e9781c1cc4532b35b9cbe8ccb: Status 404 returned error can't find the container with id 1710dc6abccc4af38e6d82e9e5d7be938ac95f7e9781c1cc4532b35b9cbe8ccb Nov 03 22:18:48.154894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.154856 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks"] Nov 03 22:18:48.159813 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.159792 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:18:48.210515 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.210472 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-906673-59486f8964-dbgks\" (UID: \"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:18:48.210657 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.210582 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xst5w\" (UniqueName: \"kubernetes.io/projected/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-kube-api-access-xst5w\") pod \"payload-processing-e2e-worker-w2-906673-59486f8964-dbgks\" (UID: \"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:18:48.257611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.257580 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-906673-dockercfg-lcq2m\"" Nov 03 22:18:48.257866 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.257847 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-906673\"" Nov 03 22:18:48.259382 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.259362 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks"] Nov 03 22:18:48.311776 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.311741 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xst5w\" (UniqueName: \"kubernetes.io/projected/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-kube-api-access-xst5w\") pod \"payload-processing-e2e-worker-w2-906673-59486f8964-dbgks\" (UID: \"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:18:48.311933 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.311890 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-906673-59486f8964-dbgks\" (UID: \"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:18:48.312559 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.312536 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-906673-59486f8964-dbgks\" (UID: \"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:18:48.366193 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.366162 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xst5w\" (UniqueName: \"kubernetes.io/projected/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-kube-api-access-xst5w\") pod \"payload-processing-e2e-worker-w2-906673-59486f8964-dbgks\" (UID: \"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:18:48.473296 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.473271 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:18:48.509713 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.509677 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" event={"ID":"7d7a8dea-f57e-40f0-8dfc-751843570fde","Type":"ContainerStarted","Data":"6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7"} Nov 03 22:18:48.509870 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.509748 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" event={"ID":"7d7a8dea-f57e-40f0-8dfc-751843570fde","Type":"ContainerStarted","Data":"1710dc6abccc4af38e6d82e9e5d7be938ac95f7e9781c1cc4532b35b9cbe8ccb"} Nov 03 22:18:48.509870 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.509783 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:48.618490 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.618440 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" podStartSLOduration=1.7603967109999998 podStartE2EDuration="2.618427283s" totalImagesPullingTime="858.030572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:48.61751722 +0000 UTC m=+648.029703238" watchObservedRunningTime="2026-11-03 22:18:48.618427283 +0000 UTC m=+648.030613299" Nov 03 22:18:48.842211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:48.842086 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks"] Nov 03 22:18:48.845163 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:18:48.845127 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735f20bd_8a1e_4c65_ac98_6bf603a6ecfa.slice/crio-3dc697ee4e4ebcc2dd0e8e7f24efcb8b81c0fecd42685d36527075bfcbe4228c WatchSource:0}: Error finding container 3dc697ee4e4ebcc2dd0e8e7f24efcb8b81c0fecd42685d36527075bfcbe4228c: Status 404 returned error can't find the container with id 3dc697ee4e4ebcc2dd0e8e7f24efcb8b81c0fecd42685d36527075bfcbe4228c Nov 03 22:18:49.485274 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:49.485245 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:18:49.515742 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:49.515708 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" event={"ID":"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa","Type":"ContainerStarted","Data":"38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290"} Nov 03 22:18:49.515936 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:49.515763 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" event={"ID":"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa","Type":"ContainerStarted","Data":"3dc697ee4e4ebcc2dd0e8e7f24efcb8b81c0fecd42685d36527075bfcbe4228c"} Nov 03 22:18:49.685841 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:49.685805 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:18:49.705977 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:49.702022 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" podStartSLOduration=1.702006684 podStartE2EDuration="1.702006684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:49.701672626 +0000 UTC m=+649.113858643" watchObservedRunningTime="2026-11-03 22:18:49.702006684 +0000 UTC m=+649.114192703" Nov 03 22:18:50.520699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.520670 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:18:50.598352 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.598321 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm"] Nov 03 22:18:50.602256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.602236 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:18:50.636596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.636566 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qlpn\" (UniqueName: \"kubernetes.io/projected/d1819764-af2f-4666-8b13-bc4bda3a1f09-kube-api-access-8qlpn\") pod \"payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm\" (UID: \"d1819764-af2f-4666-8b13-bc4bda3a1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:18:50.636845 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.636824 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1819764-af2f-4666-8b13-bc4bda3a1f09-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm\" (UID: \"d1819764-af2f-4666-8b13-bc4bda3a1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:18:50.652882 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.652859 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-6a6b7f\"" Nov 03 22:18:50.700100 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.700059 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-6a6b7f-dockercfg-dcsr2\"" Nov 03 22:18:50.707038 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.707005 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm"] Nov 03 22:18:50.737715 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.737685 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1819764-af2f-4666-8b13-bc4bda3a1f09-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm\" (UID: \"d1819764-af2f-4666-8b13-bc4bda3a1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:18:50.737896 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.737742 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qlpn\" (UniqueName: \"kubernetes.io/projected/d1819764-af2f-4666-8b13-bc4bda3a1f09-kube-api-access-8qlpn\") pod \"payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm\" (UID: \"d1819764-af2f-4666-8b13-bc4bda3a1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:18:50.738382 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.738361 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1819764-af2f-4666-8b13-bc4bda3a1f09-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm\" (UID: \"d1819764-af2f-4666-8b13-bc4bda3a1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:18:50.791329 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.791262 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qlpn\" (UniqueName: \"kubernetes.io/projected/d1819764-af2f-4666-8b13-bc4bda3a1f09-kube-api-access-8qlpn\") pod \"payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm\" (UID: \"d1819764-af2f-4666-8b13-bc4bda3a1f09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:18:50.916212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:50.916180 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:18:51.279023 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.278997 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm"] Nov 03 22:18:51.281162 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:18:51.281129 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1819764_af2f_4666_8b13_bc4bda3a1f09.slice/crio-4e169869e5c99ee5ddc7bd27f08295bc443594aca2f5002d850a99799993e497 WatchSource:0}: Error finding container 4e169869e5c99ee5ddc7bd27f08295bc443594aca2f5002d850a99799993e497: Status 404 returned error can't find the container with id 4e169869e5c99ee5ddc7bd27f08295bc443594aca2f5002d850a99799993e497 Nov 03 22:18:51.300352 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.300328 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865"] Nov 03 22:18:51.307148 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.307126 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.342629 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.342605 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-metrics-certs\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.342731 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.342654 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf62f19-94ba-4650-80a5-f19383286180-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.342771 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.342745 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-maas-api-tls\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.342880 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.342862 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzr4\" (UniqueName: \"kubernetes.io/projected/baf62f19-94ba-4650-80a5-f19383286180-kube-api-access-hnzr4\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.355893 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.355873 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-41bb2b-serving-cert\"" Nov 03 22:18:51.384150 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.384114 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865"] Nov 03 22:18:51.444317 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.444279 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-metrics-certs\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.444543 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.444354 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf62f19-94ba-4650-80a5-f19383286180-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.444543 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.444411 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-maas-api-tls\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.444689 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.444567 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzr4\" (UniqueName: \"kubernetes.io/projected/baf62f19-94ba-4650-80a5-f19383286180-kube-api-access-hnzr4\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.444746 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:18:51.444689 6234 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-serving-cert: secret "maas-api-e2e-worker-w3-41bb2b-serving-cert" not found Nov 03 22:18:51.444804 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:18:51.444758 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-maas-api-tls podName:baf62f19-94ba-4650-80a5-f19383286180 nodeName:}" failed. No retries permitted until 2026-11-03 22:18:51.944737087 +0000 UTC m=+651.356923121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-maas-api-tls") pod "maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" (UID: "baf62f19-94ba-4650-80a5-f19383286180") : secret "maas-api-e2e-worker-w3-41bb2b-serving-cert" not found Nov 03 22:18:51.445081 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.445061 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf62f19-94ba-4650-80a5-f19383286180-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.447484 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.447465 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-metrics-certs\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.497692 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.497661 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzr4\" (UniqueName: \"kubernetes.io/projected/baf62f19-94ba-4650-80a5-f19383286180-kube-api-access-hnzr4\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.526599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.526566 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" event={"ID":"d1819764-af2f-4666-8b13-bc4bda3a1f09","Type":"ContainerStarted","Data":"3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781"} Nov 03 22:18:51.526733 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.526619 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" event={"ID":"d1819764-af2f-4666-8b13-bc4bda3a1f09","Type":"ContainerStarted","Data":"4e169869e5c99ee5ddc7bd27f08295bc443594aca2f5002d850a99799993e497"} Nov 03 22:18:51.526903 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.526882 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:18:51.633662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.633556 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" podStartSLOduration=1.633544035 podStartE2EDuration="1.633544035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:51.632472167 +0000 UTC m=+651.044658181" watchObservedRunningTime="2026-11-03 22:18:51.633544035 +0000 UTC m=+651.045730052" Nov 03 22:18:51.950218 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.950189 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-maas-api-tls\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:51.952574 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:51.952550 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-maas-api-tls\") pod \"maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:52.229175 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:52.229089 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:52.594696 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:52.594670 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865"] Nov 03 22:18:52.596773 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:18:52.596745 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf62f19_94ba_4650_80a5_f19383286180.slice/crio-fc4f5e5ccddb91d297d19a17ccd44c791e3b418468a64d29cb55a04f42c36854 WatchSource:0}: Error finding container fc4f5e5ccddb91d297d19a17ccd44c791e3b418468a64d29cb55a04f42c36854: Status 404 returned error can't find the container with id fc4f5e5ccddb91d297d19a17ccd44c791e3b418468a64d29cb55a04f42c36854 Nov 03 22:18:53.538647 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:53.538616 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" event={"ID":"baf62f19-94ba-4650-80a5-f19383286180","Type":"ContainerStarted","Data":"9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463"} Nov 03 22:18:53.538979 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:53.538679 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" event={"ID":"baf62f19-94ba-4650-80a5-f19383286180","Type":"ContainerStarted","Data":"fc4f5e5ccddb91d297d19a17ccd44c791e3b418468a64d29cb55a04f42c36854"} Nov 03 22:18:53.538979 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:53.538708 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:18:53.677706 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:53.677654 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" podStartSLOduration=1.8079193359999999 podStartE2EDuration="2.677639989s" totalImagesPullingTime="869.720653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:53.677234401 +0000 UTC m=+653.089420419" watchObservedRunningTime="2026-11-03 22:18:53.677639989 +0000 UTC m=+653.089826016" Nov 03 22:18:54.521632 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:54.521608 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:18:55.490883 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:55.490855 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:18:57.506782 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:57.506744 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:18:57.507273 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:57.507025 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:18:57.507273 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:57.507108 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:18:58.996134 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:18:58.996102 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:18:59.500688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:59.500653 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:18:59.550185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:18:59.550148 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:19:00.622997 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.622966 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb"] Nov 03 22:19:00.626748 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.626727 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.675909 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.675886 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-f81f8f-serving-cert\"" Nov 03 22:19:00.724242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.724212 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb"] Nov 03 22:19:00.745210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.745177 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43255448-b9e8-411d-8c9a-d3ca73526c1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.745358 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.745295 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8gds\" (UniqueName: \"kubernetes.io/projected/43255448-b9e8-411d-8c9a-d3ca73526c1e-kube-api-access-z8gds\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.745358 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.745352 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.745454 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.745442 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-metrics-certs\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.846217 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.846180 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8gds\" (UniqueName: \"kubernetes.io/projected/43255448-b9e8-411d-8c9a-d3ca73526c1e-kube-api-access-z8gds\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.846412 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.846231 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.846412 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.846301 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-metrics-certs\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.846412 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.846344 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43255448-b9e8-411d-8c9a-d3ca73526c1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.846612 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:19:00.846438 6234 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-serving-cert: secret "maas-api-e2e-worker-w5-f81f8f-serving-cert" not found Nov 03 22:19:00.846776 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:19:00.846756 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-maas-api-tls podName:43255448-b9e8-411d-8c9a-d3ca73526c1e nodeName:}" failed. No retries permitted until 2026-11-03 22:19:01.346730142 +0000 UTC m=+660.758916153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-maas-api-tls") pod "maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" (UID: "43255448-b9e8-411d-8c9a-d3ca73526c1e") : secret "maas-api-e2e-worker-w5-f81f8f-serving-cert" not found Nov 03 22:19:00.847152 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.847131 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43255448-b9e8-411d-8c9a-d3ca73526c1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.849216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.849189 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-metrics-certs\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:00.902648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:00.902577 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8gds\" (UniqueName: \"kubernetes.io/projected/43255448-b9e8-411d-8c9a-d3ca73526c1e-kube-api-access-z8gds\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:01.351667 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:01.351630 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:01.353968 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:01.353948 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:01.529915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:01.529888 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:19:01.539407 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:01.539383 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:01.906562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:01.906511 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb"] Nov 03 22:19:01.909032 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:19:01.909003 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43255448_b9e8_411d_8c9a_d3ca73526c1e.slice/crio-6a1c2f57c220be9259ed3b4edcdfa887ef587bde7011d7a6801d4571d67165fb WatchSource:0}: Error finding container 6a1c2f57c220be9259ed3b4edcdfa887ef587bde7011d7a6801d4571d67165fb: Status 404 returned error can't find the container with id 6a1c2f57c220be9259ed3b4edcdfa887ef587bde7011d7a6801d4571d67165fb Nov 03 22:19:02.534336 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:02.534301 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:19:02.585639 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:02.585605 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" event={"ID":"43255448-b9e8-411d-8c9a-d3ca73526c1e","Type":"ContainerStarted","Data":"6a1c2f57c220be9259ed3b4edcdfa887ef587bde7011d7a6801d4571d67165fb"} Nov 03 22:19:03.593704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:03.593667 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" event={"ID":"43255448-b9e8-411d-8c9a-d3ca73526c1e","Type":"ContainerStarted","Data":"84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35"} Nov 03 22:19:03.594382 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:03.594346 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:03.703040 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:03.702988 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" podStartSLOduration=2.813416464 podStartE2EDuration="3.702974529s" totalImagesPullingTime="889.558065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:03.701400005 +0000 UTC m=+663.113586020" watchObservedRunningTime="2026-11-03 22:19:03.702974529 +0000 UTC m=+663.115160593" Nov 03 22:19:08.953465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:08.953426 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:19:09.604410 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:09.604384 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:19:10.131218 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.131180 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt"] Nov 03 22:19:10.138042 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.138016 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.188916 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.188885 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt"] Nov 03 22:19:10.189086 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.188936 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-6a6b7f-serving-cert\"" Nov 03 22:19:10.251440 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.251390 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62dlb\" (UniqueName: \"kubernetes.io/projected/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-kube-api-access-62dlb\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.251605 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.251460 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.251605 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.251544 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.251711 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.251602 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-metrics-certs\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.352666 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.352637 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.352844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.352673 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.352844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.352693 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-metrics-certs\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.352844 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:19:10.352804 6234 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-serving-cert: secret "maas-api-e2e-worker-w6-6a6b7f-serving-cert" not found Nov 03 22:19:10.352844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.352829 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dlb\" (UniqueName: \"kubernetes.io/projected/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-kube-api-access-62dlb\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.353008 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:19:10.352872 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-maas-api-tls podName:228e0c72-3449-49d8-a3c5-d96d0a8ccacc nodeName:}" failed. No retries permitted until 2026-11-03 22:19:10.852852697 +0000 UTC m=+670.265038712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-maas-api-tls") pod "maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" (UID: "228e0c72-3449-49d8-a3c5-d96d0a8ccacc") : secret "maas-api-e2e-worker-w6-6a6b7f-serving-cert" not found Nov 03 22:19:10.353390 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.353367 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.355124 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.355104 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-metrics-certs\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.408999 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.408922 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dlb\" (UniqueName: \"kubernetes.io/projected/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-kube-api-access-62dlb\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.858737 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.858701 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:10.861260 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:10.861231 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:11.051548 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:11.051515 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:11.422650 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:11.422593 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt"] Nov 03 22:19:11.633079 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:11.633045 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" event={"ID":"228e0c72-3449-49d8-a3c5-d96d0a8ccacc","Type":"ContainerStarted","Data":"201e4c52b5a16640aed394d36ce41c7939b59333c680e7c73ee2de51a0924901"} Nov 03 22:19:12.640334 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:12.640247 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" event={"ID":"228e0c72-3449-49d8-a3c5-d96d0a8ccacc","Type":"ContainerStarted","Data":"cb23c82aa301e2e3e6b5ba03b14f1787c5906890fc41b7ae917249233dd052bb"} Nov 03 22:19:12.640334 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:12.640320 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:12.749015 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:12.748963 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" podStartSLOduration=1.9069519860000002 podStartE2EDuration="2.748948508s" totalImagesPullingTime="841.996522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:12.747798471 +0000 UTC m=+672.159984500" watchObservedRunningTime="2026-11-03 22:19:12.748948508 +0000 UTC m=+672.161134524" Nov 03 22:19:18.652313 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:18.652281 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:19:19.895467 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:19.895433 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:19:19.951381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:19.951340 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm"] Nov 03 22:19:19.955457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:19.955432 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.048053 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.048019 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-home\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.048053 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.048062 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhbw\" (UniqueName: \"kubernetes.io/projected/e33680db-536e-4c35-b6ac-1765204cc302-kube-api-access-wrhbw\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.048270 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.048101 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-dshm\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.048355 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.048261 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-tmp-dir\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.048394 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.048360 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-model-cache\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.048457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.048437 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33680db-536e-4c35-b6ac-1765204cc302-tls-certs\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.053822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.053798 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"default-dockercfg-9jmx4\"" Nov 03 22:19:20.053946 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.053844 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"simulator-w1-3378fd-kserve-self-signed-certs\"" Nov 03 22:19:20.053946 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.053904 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"openshift-service-ca.crt\"" Nov 03 22:19:20.054030 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.053954 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"kube-root-ca.crt\"" Nov 03 22:19:20.055890 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.055869 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm"] Nov 03 22:19:20.149484 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.149395 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-model-cache\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.149484 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.149443 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33680db-536e-4c35-b6ac-1765204cc302-tls-certs\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.149732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.149524 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-home\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.149732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.149552 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhbw\" (UniqueName: \"kubernetes.io/projected/e33680db-536e-4c35-b6ac-1765204cc302-kube-api-access-wrhbw\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.149732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.149585 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-dshm\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.149732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.149650 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-tmp-dir\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.149928 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.149900 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-model-cache\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.149995 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.149950 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-home\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.150132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.150102 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-tmp-dir\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.151987 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.151958 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33680db-536e-4c35-b6ac-1765204cc302-tls-certs\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.152105 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.152007 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-dshm\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.203005 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.202968 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhbw\" (UniqueName: \"kubernetes.io/projected/e33680db-536e-4c35-b6ac-1765204cc302-kube-api-access-wrhbw\") pod \"simulator-w1-3378fd-kserve-55848f8dd9-6wcrm\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.266946 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.266915 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:20.634086 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.633947 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm"] Nov 03 22:19:20.636654 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:19:20.636624 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode33680db_536e_4c35_b6ac_1765204cc302.slice/crio-cf52841bff28752f6f319587614f3aacadac4bc31f2ac9b09351f16ef27ba61c WatchSource:0}: Error finding container cf52841bff28752f6f319587614f3aacadac4bc31f2ac9b09351f16ef27ba61c: Status 404 returned error can't find the container with id cf52841bff28752f6f319587614f3aacadac4bc31f2ac9b09351f16ef27ba61c Nov 03 22:19:20.687108 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:20.687082 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" event={"ID":"e33680db-536e-4c35-b6ac-1765204cc302","Type":"ContainerStarted","Data":"cf52841bff28752f6f319587614f3aacadac4bc31f2ac9b09351f16ef27ba61c"} Nov 03 22:19:21.693094 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:21.693054 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" event={"ID":"e33680db-536e-4c35-b6ac-1765204cc302","Type":"ContainerStarted","Data":"bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184"} Nov 03 22:19:21.801127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:21.801065 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podStartSLOduration=2.801048223 podStartE2EDuration="2.801048223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:21.800852459 +0000 UTC m=+681.213038477" watchObservedRunningTime="2026-11-03 22:19:21.801048223 +0000 UTC m=+681.213234240" Nov 03 22:19:22.267596 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:22.267565 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:24.730408 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.730370 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476"] Nov 03 22:19:24.734483 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.734459 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.799865 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.799826 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ccv\" (UniqueName: \"kubernetes.io/projected/016058c4-df1f-4952-970d-e3f9ffe4704d-kube-api-access-d6ccv\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.800125 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.800092 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-metrics-certs\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.800258 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.800162 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.800258 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.800211 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016058c4-df1f-4952-970d-e3f9ffe4704d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.831736 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.831705 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f7e6db-serving-cert\"" Nov 03 22:19:24.833598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.833576 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476"] Nov 03 22:19:24.901732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.901691 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-metrics-certs\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.901732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.901734 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.901997 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.901755 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016058c4-df1f-4952-970d-e3f9ffe4704d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.901997 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:19:24.901882 6234 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-serving-cert: secret "maas-api-e2e-worker-w4-f7e6db-serving-cert" not found Nov 03 22:19:24.901997 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:19:24.901952 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-maas-api-tls podName:016058c4-df1f-4952-970d-e3f9ffe4704d nodeName:}" failed. No retries permitted until 2026-11-03 22:19:25.401930132 +0000 UTC m=+684.814116127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-maas-api-tls") pod "maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" (UID: "016058c4-df1f-4952-970d-e3f9ffe4704d") : secret "maas-api-e2e-worker-w4-f7e6db-serving-cert" not found Nov 03 22:19:24.901997 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.901890 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ccv\" (UniqueName: \"kubernetes.io/projected/016058c4-df1f-4952-970d-e3f9ffe4704d-kube-api-access-d6ccv\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.902458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.902434 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016058c4-df1f-4952-970d-e3f9ffe4704d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.904095 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.904074 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-metrics-certs\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:24.955914 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:24.955889 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ccv\" (UniqueName: \"kubernetes.io/projected/016058c4-df1f-4952-970d-e3f9ffe4704d-kube-api-access-d6ccv\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:25.408264 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:25.408226 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:25.410593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:25.410569 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:25.647480 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:25.647436 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:26.013617 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:26.013596 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476"] Nov 03 22:19:26.016052 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:19:26.016015 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016058c4_df1f_4952_970d_e3f9ffe4704d.slice/crio-5aa22b36a41144629946c497988c44a25198c706a2f32d58bc174a2fc4b0d100 WatchSource:0}: Error finding container 5aa22b36a41144629946c497988c44a25198c706a2f32d58bc174a2fc4b0d100: Status 404 returned error can't find the container with id 5aa22b36a41144629946c497988c44a25198c706a2f32d58bc174a2fc4b0d100 Nov 03 22:19:26.725234 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:26.725203 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" event={"ID":"016058c4-df1f-4952-970d-e3f9ffe4704d","Type":"ContainerStarted","Data":"5aa22b36a41144629946c497988c44a25198c706a2f32d58bc174a2fc4b0d100"} Nov 03 22:19:27.732129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:27.732092 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" event={"ID":"016058c4-df1f-4952-970d-e3f9ffe4704d","Type":"ContainerStarted","Data":"5438fc35e36a4feb8d3027338d8dc2b0dddd62a12365575bcf7939a6dbe637e6"} Nov 03 22:19:27.732522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:27.732187 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:27.843088 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:27.843031 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" podStartSLOduration=3.012117789 podStartE2EDuration="3.843016688s" totalImagesPullingTime="830.898899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:27.840567452 +0000 UTC m=+687.252753467" watchObservedRunningTime="2026-11-03 22:19:27.843016688 +0000 UTC m=+687.255202698" Nov 03 22:19:28.998375 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:19:28.998341 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:19:30.267250 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:30.267211 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:30.280029 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:30.280003 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:30.756786 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:30.756753 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:19:31.423867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:31.423830 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:19:33.742145 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:33.742118 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:19:50.698334 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:19:50.698074 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:19:58.994858 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:19:58.994823 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:20:08.370707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:08.370669 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:20:11.223223 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.223142 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b"] Nov 03 22:20:11.228469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.228448 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.281767 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.281726 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b"] Nov 03 22:20:11.301666 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.301635 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-model-cache\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.301814 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.301786 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5kxt\" (UniqueName: \"kubernetes.io/projected/c0d1460a-5288-4f03-8159-34390ce36678-kube-api-access-p5kxt\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.301993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.301973 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d1460a-5288-4f03-8159-34390ce36678-tls-certs\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.302064 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.302017 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-tmp-dir\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.302064 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.302050 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-home\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.302170 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.302091 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-dshm\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.325942 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.325913 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"premium-w1-3378fd-kserve-self-signed-certs\"" Nov 03 22:20:11.402810 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.402763 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d1460a-5288-4f03-8159-34390ce36678-tls-certs\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.403004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.402820 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-tmp-dir\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.403004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.402849 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-home\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.403004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.402872 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-dshm\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.403004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.402961 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-model-cache\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.403228 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.403048 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5kxt\" (UniqueName: \"kubernetes.io/projected/c0d1460a-5288-4f03-8159-34390ce36678-kube-api-access-p5kxt\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.403371 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.403342 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-tmp-dir\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.403545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.403430 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-model-cache\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.403729 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.403710 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-home\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.405301 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.405282 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-dshm\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.407345 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.405714 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d1460a-5288-4f03-8159-34390ce36678-tls-certs\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.460158 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.460122 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5kxt\" (UniqueName: \"kubernetes.io/projected/c0d1460a-5288-4f03-8159-34390ce36678-kube-api-access-p5kxt\") pod \"premium-w1-3378fd-kserve-8474fb444-zdb8b\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.539976 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.539892 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:11.919971 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.919947 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b"] Nov 03 22:20:11.922303 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:20:11.922261 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0d1460a_5288_4f03_8159_34390ce36678.slice/crio-96170f51eddf35a74f43aacb1888569610c7944dadc6c22a3a7b144059664388 WatchSource:0}: Error finding container 96170f51eddf35a74f43aacb1888569610c7944dadc6c22a3a7b144059664388: Status 404 returned error can't find the container with id 96170f51eddf35a74f43aacb1888569610c7944dadc6c22a3a7b144059664388 Nov 03 22:20:11.963251 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:11.963220 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" event={"ID":"c0d1460a-5288-4f03-8159-34390ce36678","Type":"ContainerStarted","Data":"96170f51eddf35a74f43aacb1888569610c7944dadc6c22a3a7b144059664388"} Nov 03 22:20:12.969862 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:12.969819 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" event={"ID":"c0d1460a-5288-4f03-8159-34390ce36678","Type":"ContainerStarted","Data":"547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0"} Nov 03 22:20:13.077083 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:13.077032 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podStartSLOduration=2.077019301 podStartE2EDuration="2.077019301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:13.076002661 +0000 UTC m=+732.488188676" watchObservedRunningTime="2026-11-03 22:20:13.077019301 +0000 UTC m=+732.489205391" Nov 03 22:20:13.541063 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:13.541030 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:21.540536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:21.540488 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:21.553301 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:21.553269 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:22.027102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:22.027074 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:20:27.399763 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:27.399725 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:20:28.996060 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:20:28.996016 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:20:40.058067 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.058036 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz"] Nov 03 22:20:40.067047 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.067026 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.115023 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.114986 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz"] Nov 03 22:20:40.159574 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.159540 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"openshift-service-ca.crt\"" Nov 03 22:20:40.159872 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.159854 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"kube-root-ca.crt\"" Nov 03 22:20:40.159924 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.159892 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"simulator-w0-f61714-kserve-self-signed-certs\"" Nov 03 22:20:40.162093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.162077 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"default-dockercfg-hrt5c\"" Nov 03 22:20:40.202972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.202943 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822lc\" (UniqueName: \"kubernetes.io/projected/71a3ff55-de90-431d-b281-3f2333cb7f1c-kube-api-access-822lc\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.203117 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.202994 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-dshm\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.203168 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.203117 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-home\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.203168 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.203152 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-model-cache\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.203266 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.203173 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-tmp-dir\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.203266 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.203203 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a3ff55-de90-431d-b281-3f2333cb7f1c-tls-certs\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.304134 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.304099 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822lc\" (UniqueName: \"kubernetes.io/projected/71a3ff55-de90-431d-b281-3f2333cb7f1c-kube-api-access-822lc\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.304335 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.304147 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-dshm\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.304335 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.304213 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-home\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.304335 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.304238 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-model-cache\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.304335 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.304255 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-tmp-dir\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.304335 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.304292 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a3ff55-de90-431d-b281-3f2333cb7f1c-tls-certs\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.304638 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.304609 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-home\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.304762 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.304716 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-model-cache\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.304805 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.304760 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-tmp-dir\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.306625 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.306601 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-dshm\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.306799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.306781 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a3ff55-de90-431d-b281-3f2333cb7f1c-tls-certs\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.357777 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.357711 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-822lc\" (UniqueName: \"kubernetes.io/projected/71a3ff55-de90-431d-b281-3f2333cb7f1c-kube-api-access-822lc\") pod \"simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.382729 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.382705 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:40.746545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:40.746518 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz"] Nov 03 22:20:40.749072 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:20:40.749045 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71a3ff55_de90_431d_b281_3f2333cb7f1c.slice/crio-a478dddb0e5efb00caae4fe4ecac13b399474596f1c001b91bbbcecf024ffc81 WatchSource:0}: Error finding container a478dddb0e5efb00caae4fe4ecac13b399474596f1c001b91bbbcecf024ffc81: Status 404 returned error can't find the container with id a478dddb0e5efb00caae4fe4ecac13b399474596f1c001b91bbbcecf024ffc81 Nov 03 22:20:41.118954 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:41.118864 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" event={"ID":"71a3ff55-de90-431d-b281-3f2333cb7f1c","Type":"ContainerStarted","Data":"a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b"} Nov 03 22:20:41.118954 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:41.118905 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" event={"ID":"71a3ff55-de90-431d-b281-3f2333cb7f1c","Type":"ContainerStarted","Data":"a478dddb0e5efb00caae4fe4ecac13b399474596f1c001b91bbbcecf024ffc81"} Nov 03 22:20:41.227874 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:41.227820 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podStartSLOduration=1.227806391 podStartE2EDuration="1.227806391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:41.226676595 +0000 UTC m=+760.638862593" watchObservedRunningTime="2026-11-03 22:20:41.227806391 +0000 UTC m=+760.639992407" Nov 03 22:20:41.383252 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:41.383174 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:46.615257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:46.615219 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:20:50.383068 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:50.383040 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:50.396889 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:50.396858 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:51.181485 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:51.181453 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:20:53.714733 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.714701 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26"] Nov 03 22:20:53.718488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.718467 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.815797 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.815767 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f7e6db\"/\"simulator-w4-f7e6db-kserve-self-signed-certs\"" Nov 03 22:20:53.815797 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.815793 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f7e6db\"/\"kube-root-ca.crt\"" Nov 03 22:20:53.816005 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.815806 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f7e6db\"/\"openshift-service-ca.crt\"" Nov 03 22:20:53.816005 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.815891 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f7e6db\"/\"default-dockercfg-xdf7f\"" Nov 03 22:20:53.817580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.817561 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26"] Nov 03 22:20:53.852908 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.852883 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-home\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.853039 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.852994 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tmp-dir\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.853039 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.853015 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-dshm\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.853139 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.853082 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-model-cache\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.853184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.853145 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lswkh\" (UniqueName: \"kubernetes.io/projected/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-kube-api-access-lswkh\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.853184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.853173 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tls-certs\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.954754 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.954722 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tmp-dir\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.954754 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.954755 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-dshm\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.954993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.954776 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-model-cache\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.954993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.954824 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lswkh\" (UniqueName: \"kubernetes.io/projected/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-kube-api-access-lswkh\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.954993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.954861 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tls-certs\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.954993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.954923 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-home\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.955259 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.955169 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tmp-dir\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.955259 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.955225 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-model-cache\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.955331 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.955270 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-home\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.957172 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.957148 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-dshm\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:53.957375 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:53.957359 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tls-certs\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:54.007890 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:54.007822 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lswkh\" (UniqueName: \"kubernetes.io/projected/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-kube-api-access-lswkh\") pod \"simulator-w4-f7e6db-kserve-6587df6689-s8j26\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:54.030599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:54.030575 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:54.395433 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:54.395409 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26"] Nov 03 22:20:54.397478 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:20:54.397410 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec6adb50_5a16_44e8_9f09_a45b32cafd1f.slice/crio-4a6e52e5561cfb2bd3b5d7453cacca5d6f24edb268b2689e5c7e3730eb9fa4d4 WatchSource:0}: Error finding container 4a6e52e5561cfb2bd3b5d7453cacca5d6f24edb268b2689e5c7e3730eb9fa4d4: Status 404 returned error can't find the container with id 4a6e52e5561cfb2bd3b5d7453cacca5d6f24edb268b2689e5c7e3730eb9fa4d4 Nov 03 22:20:55.202843 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:55.202807 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" event={"ID":"ec6adb50-5a16-44e8-9f09-a45b32cafd1f","Type":"ContainerStarted","Data":"a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f"} Nov 03 22:20:55.202843 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:55.202848 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" event={"ID":"ec6adb50-5a16-44e8-9f09-a45b32cafd1f","Type":"ContainerStarted","Data":"4a6e52e5561cfb2bd3b5d7453cacca5d6f24edb268b2689e5c7e3730eb9fa4d4"} Nov 03 22:20:55.310485 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:55.310433 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podStartSLOduration=2.310418941 podStartE2EDuration="2.310418941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:55.309663122 +0000 UTC m=+774.721849165" watchObservedRunningTime="2026-11-03 22:20:55.310418941 +0000 UTC m=+774.722604957" Nov 03 22:20:56.030835 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:20:56.030802 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:20:58.995939 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:20:58.995906 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:21:04.031466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:04.031428 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:21:04.044037 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:04.044009 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:21:04.278454 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:04.278426 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:21:11.864671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.864639 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph"] Nov 03 22:21:11.879773 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.879738 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:11.922000 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.921970 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph"] Nov 03 22:21:11.948263 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.948232 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-home\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:11.948405 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.948280 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsnht\" (UniqueName: \"kubernetes.io/projected/6866d963-bd92-4452-be98-9bdd6a7e9e11-kube-api-access-hsnht\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:11.948405 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.948350 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-dshm\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:11.948477 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.948427 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-tmp-dir\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:11.948525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.948486 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-model-cache\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:11.948566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.948540 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6866d963-bd92-4452-be98-9bdd6a7e9e11-tls-certs\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:11.966106 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.966067 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"premium-w3-41bb2b-kserve-self-signed-certs\"" Nov 03 22:21:11.966249 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.966127 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"openshift-service-ca.crt\"" Nov 03 22:21:11.966249 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.966128 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"default-dockercfg-kdmdn\"" Nov 03 22:21:11.975085 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:11.975070 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"kube-root-ca.crt\"" Nov 03 22:21:12.049764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.049730 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-home\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.049930 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.049774 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsnht\" (UniqueName: \"kubernetes.io/projected/6866d963-bd92-4452-be98-9bdd6a7e9e11-kube-api-access-hsnht\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.049930 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.049890 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-dshm\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.050014 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.049940 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-tmp-dir\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.050014 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.050000 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-model-cache\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.050103 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.050030 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6866d963-bd92-4452-be98-9bdd6a7e9e11-tls-certs\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.050213 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.050191 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-home\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.050295 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.050279 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-tmp-dir\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.050383 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.050361 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-model-cache\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.052138 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.052116 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-dshm\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.052374 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.052356 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6866d963-bd92-4452-be98-9bdd6a7e9e11-tls-certs\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.102726 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.102697 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsnht\" (UniqueName: \"kubernetes.io/projected/6866d963-bd92-4452-be98-9bdd6a7e9e11-kube-api-access-hsnht\") pod \"premium-w3-41bb2b-kserve-779f8cfc8b-rpmph\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.191218 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.191193 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:12.556506 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:12.556466 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph"] Nov 03 22:21:12.559259 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:21:12.559228 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6866d963_bd92_4452_be98_9bdd6a7e9e11.slice/crio-4d104854f7c24febcefd45c33b3ef28a1fe95cc66d4bb506b3422d7036f4e8df WatchSource:0}: Error finding container 4d104854f7c24febcefd45c33b3ef28a1fe95cc66d4bb506b3422d7036f4e8df: Status 404 returned error can't find the container with id 4d104854f7c24febcefd45c33b3ef28a1fe95cc66d4bb506b3422d7036f4e8df Nov 03 22:21:13.316811 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:13.316778 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" event={"ID":"6866d963-bd92-4452-be98-9bdd6a7e9e11","Type":"ContainerStarted","Data":"05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2"} Nov 03 22:21:13.317200 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:13.316824 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" event={"ID":"6866d963-bd92-4452-be98-9bdd6a7e9e11","Type":"ContainerStarted","Data":"4d104854f7c24febcefd45c33b3ef28a1fe95cc66d4bb506b3422d7036f4e8df"} Nov 03 22:21:13.424105 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:13.424057 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podStartSLOduration=2.424043534 podStartE2EDuration="2.424043534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:13.423442735 +0000 UTC m=+792.835628751" watchObservedRunningTime="2026-11-03 22:21:13.424043534 +0000 UTC m=+792.836229548" Nov 03 22:21:14.192030 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:14.191996 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:19.323146 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:19.323111 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:21:22.191513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:22.191473 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:22.204053 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:22.204034 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:22.373726 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:22.373696 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:21:28.996555 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:21:28.996518 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:21:52.473074 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:21:52.473029 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:21:58.996144 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:21:58.996106 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:22:01.531954 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.531920 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx"] Nov 03 22:22:01.536131 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.536105 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.589528 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.589481 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx"] Nov 03 22:22:01.633279 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.633247 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"openshift-service-ca.crt\"" Nov 03 22:22:01.633487 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.633381 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"default-dockercfg-fr8gq\"" Nov 03 22:22:01.633487 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.633390 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"kube-root-ca.crt\"" Nov 03 22:22:01.633646 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.633588 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"premium-w6-6a6b7f-kserve-self-signed-certs\"" Nov 03 22:22:01.678848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.678821 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-home\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.678969 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.678895 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrcc\" (UniqueName: \"kubernetes.io/projected/7df4efee-bc6e-4a65-b97b-5228c8a9d408-kube-api-access-2jrcc\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.679021 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.679001 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-model-cache\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.679076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.679060 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-dshm\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.679152 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.679134 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tls-certs\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.679187 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.679172 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tmp-dir\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.780030 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.779998 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-dshm\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.780199 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.780037 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tls-certs\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.780199 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.780070 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tmp-dir\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.780199 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.780130 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-home\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.780199 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.780193 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrcc\" (UniqueName: \"kubernetes.io/projected/7df4efee-bc6e-4a65-b97b-5228c8a9d408-kube-api-access-2jrcc\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.780366 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.780228 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-model-cache\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.780576 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.780552 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tmp-dir\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.780648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.780608 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-home\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.780648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.780624 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-model-cache\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.782308 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.782261 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-dshm\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.782580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.782561 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tls-certs\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.834628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.834602 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrcc\" (UniqueName: \"kubernetes.io/projected/7df4efee-bc6e-4a65-b97b-5228c8a9d408-kube-api-access-2jrcc\") pod \"premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:01.848741 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:01.848719 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:02.216400 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:02.216375 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx"] Nov 03 22:22:02.218411 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:22:02.218375 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7df4efee_bc6e_4a65_b97b_5228c8a9d408.slice/crio-33aa05f9d909d87fe1e681bb48d7e2813a5d522f0a1c4d0e94295c77b7c93955 WatchSource:0}: Error finding container 33aa05f9d909d87fe1e681bb48d7e2813a5d522f0a1c4d0e94295c77b7c93955: Status 404 returned error can't find the container with id 33aa05f9d909d87fe1e681bb48d7e2813a5d522f0a1c4d0e94295c77b7c93955 Nov 03 22:22:02.588732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:02.588649 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" event={"ID":"7df4efee-bc6e-4a65-b97b-5228c8a9d408","Type":"ContainerStarted","Data":"7be4254ab1ebfe2502ec4830b2fa1ca30287546a27fdc258d4df55e9547df3eb"} Nov 03 22:22:02.588732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:02.588692 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" event={"ID":"7df4efee-bc6e-4a65-b97b-5228c8a9d408","Type":"ContainerStarted","Data":"33aa05f9d909d87fe1e681bb48d7e2813a5d522f0a1c4d0e94295c77b7c93955"} Nov 03 22:22:02.696646 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:02.696593 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podStartSLOduration=1.696580661 podStartE2EDuration="1.696580661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:02.695203259 +0000 UTC m=+842.107389277" watchObservedRunningTime="2026-11-03 22:22:02.696580661 +0000 UTC m=+842.108766678" Nov 03 22:22:02.849947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:02.849866 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:11.849270 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:11.849241 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:11.861693 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:11.861668 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:12.652894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:12.652863 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:22:13.073294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:13.073259 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:22:21.408758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.408722 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph"] Nov 03 22:22:21.413029 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.413006 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.466390 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.466351 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph"] Nov 03 22:22:21.504709 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.504676 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tmp-dir\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.504863 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.504731 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-model-cache\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.504863 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.504837 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-dshm\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.504992 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.504869 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tls-certs\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.505048 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.505003 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz9tm\" (UniqueName: \"kubernetes.io/projected/bb140cc5-3a6c-4b91-aadc-d803c46a9374-kube-api-access-xz9tm\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.505104 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.505072 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-home\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.511067 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.511043 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f7e6db\"/\"premium-w4-f7e6db-kserve-self-signed-certs\"" Nov 03 22:22:21.605911 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.605868 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-model-cache\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.606092 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.605928 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-dshm\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.606092 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.605946 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tls-certs\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.606092 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.606017 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9tm\" (UniqueName: \"kubernetes.io/projected/bb140cc5-3a6c-4b91-aadc-d803c46a9374-kube-api-access-xz9tm\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.606092 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.606054 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-home\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.606312 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.606217 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tmp-dir\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.606312 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.606297 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-model-cache\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.606435 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.606411 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-home\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.606572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.606555 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tmp-dir\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.608161 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.608137 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-dshm\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.608409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.608393 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tls-certs\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.659538 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.659470 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9tm\" (UniqueName: \"kubernetes.io/projected/bb140cc5-3a6c-4b91-aadc-d803c46a9374-kube-api-access-xz9tm\") pod \"premium-w4-f7e6db-kserve-76c4ff9964-hc2ph\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:21.726209 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:21.726168 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:22.090136 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:22.090105 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph"] Nov 03 22:22:22.092702 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:22:22.092667 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb140cc5_3a6c_4b91_aadc_d803c46a9374.slice/crio-93f91026c766e8def891b99d6ea61c1608611e3d933f869c44b1cc45a69eb9e3 WatchSource:0}: Error finding container 93f91026c766e8def891b99d6ea61c1608611e3d933f869c44b1cc45a69eb9e3: Status 404 returned error can't find the container with id 93f91026c766e8def891b99d6ea61c1608611e3d933f869c44b1cc45a69eb9e3 Nov 03 22:22:22.696208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:22.696173 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" event={"ID":"bb140cc5-3a6c-4b91-aadc-d803c46a9374","Type":"ContainerStarted","Data":"27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce"} Nov 03 22:22:22.696208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:22.696217 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" event={"ID":"bb140cc5-3a6c-4b91-aadc-d803c46a9374","Type":"ContainerStarted","Data":"93f91026c766e8def891b99d6ea61c1608611e3d933f869c44b1cc45a69eb9e3"} Nov 03 22:22:22.726302 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:22.726257 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:22.806419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:22.806366 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podStartSLOduration=1.806353165 podStartE2EDuration="1.806353165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:22.805092114 +0000 UTC m=+862.217278132" watchObservedRunningTime="2026-11-03 22:22:22.806353165 +0000 UTC m=+862.218539182" Nov 03 22:22:28.996477 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:22:28.996438 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:22:31.726783 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:31.726755 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:31.739267 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:31.739236 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:31.764799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:31.764766 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:22:50.706564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:50.706529 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:22:57.025565 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:57.025531 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g"] Nov 03 22:22:57.026070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:57.025766 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerName="istio-proxy" containerID="cri-o://2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714" gracePeriod=30 Nov 03 22:22:57.072556 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:57.072523 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt"] Nov 03 22:22:57.072751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:22:57.072730 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerName="istio-proxy" containerID="cri-o://3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb" gracePeriod=30 Nov 03 22:22:58.996594 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:22:58.996555 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:23:02.332463 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.332435 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:23:02.368491 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.368465 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:23:02.371602 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.371579 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istiod-ca-cert\") pod \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " Nov 03 22:23:02.371735 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.371615 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-credential-socket\") pod \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " Nov 03 22:23:02.371735 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.371635 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-socket\") pod \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " Nov 03 22:23:02.371735 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.371677 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-podinfo\") pod \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " Nov 03 22:23:02.371838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.371750 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-envoy\") pod \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " Nov 03 22:23:02.371838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.371784 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-token\" (UniqueName: \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-token\") pod \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " Nov 03 22:23:02.371990 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.371917 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-certs\") pod \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " Nov 03 22:23:02.371990 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.371942 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-credential-socket" pod "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" (UID: "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:02.371990 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.371949 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istiod-ca-cert" pod "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" (UID: "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:02.371990 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.371955 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-data\") pod \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " Nov 03 22:23:02.372207 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.372035 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-kube-api-access-pxqh7\" (UniqueName: \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-kube-api-access-pxqh7\") pod \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\" (UID: \"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac\") " Nov 03 22:23:02.372207 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.372040 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-socket" pod "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" (UID: "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:02.372207 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.372178 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-data" pod "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" (UID: "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:02.372461 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.372438 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-certs" pod "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" (UID: "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:02.372598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.372580 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.372679 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.372603 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.372679 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.372617 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.372679 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.372631 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.372679 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.372645 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.374163 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.374140 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-podinfo" pod "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" (UID: "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:23:02.374251 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.374161 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-token" pod "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" (UID: "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:02.374251 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.374140 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-envoy" pod "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" (UID: "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:02.374469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.374455 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-kube-api-access-pxqh7" pod "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" (UID: "07a7dfc0-3a48-49cf-8e05-e1b9799f08ac"). InnerVolumeSpecName "kube-api-access-pxqh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:02.473232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473198 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-data\") pod \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " Nov 03 22:23:02.473232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473238 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-credential-socket\") pod \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " Nov 03 22:23:02.473457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473265 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-envoy\") pod \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " Nov 03 22:23:02.473457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473306 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-token\" (UniqueName: \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-token\") pod \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " Nov 03 22:23:02.473457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473357 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-certs\") pod \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " Nov 03 22:23:02.473457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473452 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-kube-api-access-shhbb\" (UniqueName: \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-kube-api-access-shhbb\") pod \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " Nov 03 22:23:02.473698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473476 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-credential-socket" pod "526cc03d-8f45-42a9-b063-f7884ffbf8fe" (UID: "526cc03d-8f45-42a9-b063-f7884ffbf8fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:02.473698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473519 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-data" pod "526cc03d-8f45-42a9-b063-f7884ffbf8fe" (UID: "526cc03d-8f45-42a9-b063-f7884ffbf8fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:02.473698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473545 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istiod-ca-cert\") pod \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " Nov 03 22:23:02.473698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473590 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-podinfo\") pod \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " Nov 03 22:23:02.473698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473641 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-socket\") pod \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\" (UID: \"526cc03d-8f45-42a9-b063-f7884ffbf8fe\") " Nov 03 22:23:02.473698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473685 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-certs" pod "526cc03d-8f45-42a9-b063-f7884ffbf8fe" (UID: "526cc03d-8f45-42a9-b063-f7884ffbf8fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:02.474039 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473812 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istiod-ca-cert" pod "526cc03d-8f45-42a9-b063-f7884ffbf8fe" (UID: "526cc03d-8f45-42a9-b063-f7884ffbf8fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:02.474039 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.473926 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-socket" pod "526cc03d-8f45-42a9-b063-f7884ffbf8fe" (UID: "526cc03d-8f45-42a9-b063-f7884ffbf8fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:02.474165 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.474146 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.474242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.474168 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.474242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.474184 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.474242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.474198 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.474242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.474211 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.474242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.474225 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxqh7\" (UniqueName: \"kubernetes.io/projected/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-kube-api-access-pxqh7\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.474242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.474240 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.474603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.474252 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.474603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.474265 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.475707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.475682 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-envoy" pod "526cc03d-8f45-42a9-b063-f7884ffbf8fe" (UID: "526cc03d-8f45-42a9-b063-f7884ffbf8fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:02.475813 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.475737 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-kube-api-access-shhbb" pod "526cc03d-8f45-42a9-b063-f7884ffbf8fe" (UID: "526cc03d-8f45-42a9-b063-f7884ffbf8fe"). InnerVolumeSpecName "kube-api-access-shhbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:02.475915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.475889 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-token" pod "526cc03d-8f45-42a9-b063-f7884ffbf8fe" (UID: "526cc03d-8f45-42a9-b063-f7884ffbf8fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:02.475915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.475903 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-podinfo" pod "526cc03d-8f45-42a9-b063-f7884ffbf8fe" (UID: "526cc03d-8f45-42a9-b063-f7884ffbf8fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:23:02.575141 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.575067 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.575141 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.575094 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.575141 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.575105 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhbb\" (UniqueName: \"kubernetes.io/projected/526cc03d-8f45-42a9-b063-f7884ffbf8fe-kube-api-access-shhbb\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.575141 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.575115 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/526cc03d-8f45-42a9-b063-f7884ffbf8fe-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:02.935518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.935463 6234 generic.go:356] "Generic (PLEG): container finished" podID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerID="2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714" exitCode=0 Nov 03 22:23:02.935678 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.935550 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" event={"ID":"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac","Type":"ContainerDied","Data":"2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714"} Nov 03 22:23:02.935678 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.935609 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" event={"ID":"07a7dfc0-3a48-49cf-8e05-e1b9799f08ac","Type":"ContainerDied","Data":"34af7a863a515da534e11b22911fd76e05b6b9a995620020e674fe9e458d9d65"} Nov 03 22:23:02.935678 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.935637 6234 scope.go:122] "RemoveContainer" containerID="2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714" Nov 03 22:23:02.935678 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.935557 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g" Nov 03 22:23:02.937258 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.937230 6234 generic.go:356] "Generic (PLEG): container finished" podID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerID="3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb" exitCode=0 Nov 03 22:23:02.937364 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.937280 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" Nov 03 22:23:02.937364 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.937302 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" event={"ID":"526cc03d-8f45-42a9-b063-f7884ffbf8fe","Type":"ContainerDied","Data":"3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb"} Nov 03 22:23:02.937364 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.937344 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt" event={"ID":"526cc03d-8f45-42a9-b063-f7884ffbf8fe","Type":"ContainerDied","Data":"87bb2f282574ce3050ff8900cf553d297d0aef492d914763b21bd4a85deeb29c"} Nov 03 22:23:02.946938 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.946902 6234 scope.go:122] "RemoveContainer" containerID="2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714" Nov 03 22:23:02.947200 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:23:02.947184 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714\": container with ID starting with 2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714 not found: ID does not exist" containerID="2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714" Nov 03 22:23:02.947280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.947211 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714"} err="failed to get container status \"2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714\": rpc error: code = NotFound desc = could not find container \"2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714\": container with ID starting with 2cbfa1b1e85490d040930c949b446787e35efd9c35d418b5c2a74062583d5714 not found: ID does not exist" Nov 03 22:23:02.947280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.947238 6234 scope.go:122] "RemoveContainer" containerID="3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb" Nov 03 22:23:02.955404 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.955388 6234 scope.go:122] "RemoveContainer" containerID="3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb" Nov 03 22:23:02.955682 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:23:02.955665 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb\": container with ID starting with 3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb not found: ID does not exist" containerID="3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb" Nov 03 22:23:02.955732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:02.955689 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb"} err="failed to get container status \"3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb\": rpc error: code = NotFound desc = could not find container \"3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb\": container with ID starting with 3697b5bd941974dd54d6085dd5160b4fa9f68662986df09567c1d932653a9bbb not found: ID does not exist" Nov 03 22:23:03.088968 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:03.088940 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt"] Nov 03 22:23:03.135739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:03.135703 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-rjhlt"] Nov 03 22:23:03.237892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:03.237823 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g"] Nov 03 22:23:03.242371 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:03.242346 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-ghb4g"] Nov 03 22:23:04.704954 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:04.704920 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" path="/var/lib/kubelet/pods/07a7dfc0-3a48-49cf-8e05-e1b9799f08ac/volumes" Nov 03 22:23:04.705370 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:04.705356 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" path="/var/lib/kubelet/pods/526cc03d-8f45-42a9-b063-f7884ffbf8fe/volumes" Nov 03 22:23:18.504078 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.504000 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc"] Nov 03 22:23:18.504541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.504474 6234 cpu_manager.go:414] "containerMap: removing container" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerName="istio-proxy" Nov 03 22:23:18.504541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.504485 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerName="istio-proxy" Nov 03 22:23:18.504541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.504529 6234 cpu_manager.go:414] "containerMap: removing container" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerName="istio-proxy" Nov 03 22:23:18.504541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.504535 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerName="istio-proxy" Nov 03 22:23:18.504685 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.504607 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a7dfc0-3a48-49cf-8e05-e1b9799f08ac" containerName="istio-proxy" Nov 03 22:23:18.504685 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.504620 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="526cc03d-8f45-42a9-b063-f7884ffbf8fe" containerName="istio-proxy" Nov 03 22:23:18.507761 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.507739 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.556930 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.556905 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f81f8f-openshift-default-dockercfg-mvg86\"" Nov 03 22:23:18.606680 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.606642 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc"] Nov 03 22:23:18.649313 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.649281 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae0e472-814e-405c-834c-1ab7a627652c-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.649478 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.649324 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.649478 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.649412 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.649575 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.649484 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.649575 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.649567 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.649638 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.649597 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae0e472-814e-405c-834c-1ab7a627652c-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.649638 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.649628 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.649701 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.649676 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbxzs\" (UniqueName: \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-kube-api-access-pbxzs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.649785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.649772 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750276 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750244 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750294 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750327 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae0e472-814e-405c-834c-1ab7a627652c-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750364 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750387 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbxzs\" (UniqueName: \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-kube-api-access-pbxzs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750453 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750786 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750514 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae0e472-814e-405c-834c-1ab7a627652c-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750786 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750553 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750786 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750598 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750943 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750800 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.750943 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.750891 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.751114 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.751092 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.751196 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.751170 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae0e472-814e-405c-834c-1ab7a627652c-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.751302 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.751282 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.752994 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.752971 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.753237 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.753215 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae0e472-814e-405c-834c-1ab7a627652c-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.803482 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.803416 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.803606 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.803566 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbxzs\" (UniqueName: \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-kube-api-access-pbxzs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:18.821422 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:18.821397 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:19.237150 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:19.237124 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc"] Nov 03 22:23:19.239209 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:23:19.239180 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae0e472_814e_405c_834c_1ab7a627652c.slice/crio-091fbe77dd2d34a314cf16d9c19b61a4217714896c16837cc556f4edcef97473 WatchSource:0}: Error finding container 091fbe77dd2d34a314cf16d9c19b61a4217714896c16837cc556f4edcef97473: Status 404 returned error can't find the container with id 091fbe77dd2d34a314cf16d9c19b61a4217714896c16837cc556f4edcef97473 Nov 03 22:23:19.241485 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:19.241453 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:23:19.241588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:19.241538 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:23:19.241588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:19.241573 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:23:20.036966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:20.036917 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" event={"ID":"cae0e472-814e-405c-834c-1ab7a627652c","Type":"ContainerStarted","Data":"15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac"} Nov 03 22:23:20.037478 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:20.036983 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" event={"ID":"cae0e472-814e-405c-834c-1ab7a627652c","Type":"ContainerStarted","Data":"091fbe77dd2d34a314cf16d9c19b61a4217714896c16837cc556f4edcef97473"} Nov 03 22:23:20.188505 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:20.188438 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" podStartSLOduration=2.188421242 podStartE2EDuration="2.188421242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:20.186120397 +0000 UTC m=+919.598306416" watchObservedRunningTime="2026-11-03 22:23:20.188421242 +0000 UTC m=+919.600607258" Nov 03 22:23:20.822303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:20.822270 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:20.823997 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:20.823969 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:23:20.824161 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:20.824034 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" podUID="cae0e472-814e-405c-834c-1ab7a627652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 22:23:21.822136 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:21.822093 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:23:21.822666 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:21.822144 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" podUID="cae0e472-814e-405c-834c-1ab7a627652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 22:23:22.822652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:22.822611 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:23:22.823036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:22.822678 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" podUID="cae0e472-814e-405c-834c-1ab7a627652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 22:23:23.822273 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:23.822237 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:23:23.822452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:23.822299 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" podUID="cae0e472-814e-405c-834c-1ab7a627652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 22:23:24.821837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:24.821804 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:23:24.822246 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:24.821868 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" podUID="cae0e472-814e-405c-834c-1ab7a627652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 22:23:25.822200 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:25.822155 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:23:25.822609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:25.822243 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" podUID="cae0e472-814e-405c-834c-1ab7a627652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 22:23:26.826696 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:26.826658 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:26.827143 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:26.827112 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:26.827859 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:26.827837 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:23:28.996447 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:23:28.996411 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:23:38.755440 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.755404 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x"] Nov 03 22:23:38.759789 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.759767 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.812293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.812256 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x"] Nov 03 22:23:38.856015 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.855981 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6a6b7f\"/\"distinct-w6-6a6b7f-kserve-self-signed-certs\"" Nov 03 22:23:38.872536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.872481 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tmp-dir\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.872725 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.872629 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhtr\" (UniqueName: \"kubernetes.io/projected/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-kube-api-access-mkhtr\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.872725 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.872662 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-model-cache\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.872813 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.872792 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tls-certs\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.872855 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.872826 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-dshm\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.872994 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.872979 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-home\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.973830 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.973796 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhtr\" (UniqueName: \"kubernetes.io/projected/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-kube-api-access-mkhtr\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.973830 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.973833 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-model-cache\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.974098 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.973890 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tls-certs\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.974098 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.973912 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-dshm\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.974098 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.973990 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-home\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.974098 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.974021 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tmp-dir\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.974391 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.974363 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-model-cache\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.974466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.974413 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tmp-dir\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.974466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.974443 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-home\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.976241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.976214 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-dshm\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:38.976610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:38.976589 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tls-certs\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:39.032358 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:39.032293 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhtr\" (UniqueName: \"kubernetes.io/projected/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-kube-api-access-mkhtr\") pod \"distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:39.074231 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:39.074198 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:39.446210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:39.446180 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x"] Nov 03 22:23:39.448529 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:23:39.448480 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99b65a57_f9eb_40e0_989d_7655c3cc3d6e.slice/crio-da43171faa15d0b874925ca0d14016560d4b7e6db2c5a60812de2d63f2c9a32c WatchSource:0}: Error finding container da43171faa15d0b874925ca0d14016560d4b7e6db2c5a60812de2d63f2c9a32c: Status 404 returned error can't find the container with id da43171faa15d0b874925ca0d14016560d4b7e6db2c5a60812de2d63f2c9a32c Nov 03 22:23:40.179843 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:40.179807 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" event={"ID":"99b65a57-f9eb-40e0-989d-7655c3cc3d6e","Type":"ContainerStarted","Data":"8c22107d623634afe4ec674382669f6a9212c15454272ce640999ba1a55218d1"} Nov 03 22:23:40.180249 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:40.179854 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" event={"ID":"99b65a57-f9eb-40e0-989d-7655c3cc3d6e","Type":"ContainerStarted","Data":"da43171faa15d0b874925ca0d14016560d4b7e6db2c5a60812de2d63f2c9a32c"} Nov 03 22:23:40.286366 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:40.286317 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podStartSLOduration=2.286305194 podStartE2EDuration="2.286305194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:40.286081463 +0000 UTC m=+939.698267482" watchObservedRunningTime="2026-11-03 22:23:40.286305194 +0000 UTC m=+939.698491210" Nov 03 22:23:41.074642 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:41.074610 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:43.624909 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:43.624876 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:23:49.074795 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:49.074762 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:49.087381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:49.087359 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:49.250891 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:23:49.250853 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:23:58.995981 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:23:58.995945 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:24:20.194811 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.194774 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8"] Nov 03 22:24:20.198764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.198743 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.247752 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.247722 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-41bb2b-openshift-default-dockercfg-kqcr4\"" Nov 03 22:24:20.297011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.296980 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8"] Nov 03 22:24:20.342324 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.342291 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.342475 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.342328 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.342475 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.342357 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.342582 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.342468 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.342582 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.342531 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2bf96-b555-484f-b589-f3e68ab2878d-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.342647 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.342628 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.342680 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.342668 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.342716 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.342700 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.342798 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.342784 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2vv\" (UniqueName: \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-kube-api-access-4p2vv\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.443941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.443908 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.443941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.443942 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444165 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.443964 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444165 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444012 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444165 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444059 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2bf96-b555-484f-b589-f3e68ab2878d-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444165 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444136 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444189 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444235 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444316 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2vv\" (UniqueName: \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-kube-api-access-4p2vv\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444409 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444469 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444642 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444597 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444642 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444626 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.444918 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.444867 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2bf96-b555-484f-b589-f3e68ab2878d-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.446529 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.446487 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.446631 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.446594 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.497741 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.497710 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.497859 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.497715 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2vv\" (UniqueName: \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-kube-api-access-4p2vv\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.511678 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.511657 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:20.927437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.927410 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8"] Nov 03 22:24:20.930102 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:24:20.930070 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a2bf96_b555_484f_b589_f3e68ab2878d.slice/crio-9c0615b2090f47997376344aedb79a335ce584c708fdc59fea3c3431b55f5ea7 WatchSource:0}: Error finding container 9c0615b2090f47997376344aedb79a335ce584c708fdc59fea3c3431b55f5ea7: Status 404 returned error can't find the container with id 9c0615b2090f47997376344aedb79a335ce584c708fdc59fea3c3431b55f5ea7 Nov 03 22:24:20.932480 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.932452 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:24:20.932581 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.932549 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:24:20.932620 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:20.932579 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:24:21.414207 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:21.414164 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" event={"ID":"a7a2bf96-b555-484f-b589-f3e68ab2878d","Type":"ContainerStarted","Data":"157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b"} Nov 03 22:24:21.414739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:21.414222 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" event={"ID":"a7a2bf96-b555-484f-b589-f3e68ab2878d","Type":"ContainerStarted","Data":"9c0615b2090f47997376344aedb79a335ce584c708fdc59fea3c3431b55f5ea7"} Nov 03 22:24:21.511872 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:21.511837 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:21.513599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:21.513565 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:21.513731 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:21.513620 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:21.526898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:21.526853 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" podStartSLOduration=1.526839895 podStartE2EDuration="1.526839895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:21.524905216 +0000 UTC m=+980.937091234" watchObservedRunningTime="2026-11-03 22:24:21.526839895 +0000 UTC m=+980.939025911" Nov 03 22:24:22.512070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:22.512036 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:22.512460 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:22.512094 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:23.512252 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:23.512216 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:23.512772 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:23.512271 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:24.512247 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:24.512210 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:24.512454 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:24.512295 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:25.512699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:25.512658 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:25.513203 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:25.512721 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:26.512237 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:26.512199 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 22:24:26.512446 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:26.512256 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 22:24:27.561249 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:27.561212 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:27.561736 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:27.561713 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:27.562559 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:27.562533 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:24:28.851171 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:24:28.851139 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:24:28.996373 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:24:28.996338 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:24:58.995535 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:24:58.995480 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:25:12.796871 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.796837 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr"] Nov 03 22:25:12.800819 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.800796 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.841043 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.841010 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca0821f-278d-456f-82e9-31578aa0b3a6-tls-certs\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.841191 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.841155 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx422\" (UniqueName: \"kubernetes.io/projected/4ca0821f-278d-456f-82e9-31578aa0b3a6-kube-api-access-vx422\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.841243 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.841229 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-home\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.841285 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.841260 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-dshm\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.841448 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.841432 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-tmp-dir\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.841521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.841509 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-model-cache\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.897592 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.897554 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"distinct-w3-41bb2b-kserve-self-signed-certs\"" Nov 03 22:25:12.899542 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.899520 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr"] Nov 03 22:25:12.942773 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.942732 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx422\" (UniqueName: \"kubernetes.io/projected/4ca0821f-278d-456f-82e9-31578aa0b3a6-kube-api-access-vx422\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.942773 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.942780 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-home\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.943017 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.942802 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-dshm\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.943017 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.942881 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-tmp-dir\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.943017 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.942916 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-model-cache\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.943017 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.942945 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca0821f-278d-456f-82e9-31578aa0b3a6-tls-certs\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.943291 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.943264 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-home\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.943524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.943345 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-tmp-dir\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.943624 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.943401 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-model-cache\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.945584 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.945563 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca0821f-278d-456f-82e9-31578aa0b3a6-tls-certs\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.945792 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.945771 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-dshm\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:12.996876 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:12.996832 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx422\" (UniqueName: \"kubernetes.io/projected/4ca0821f-278d-456f-82e9-31578aa0b3a6-kube-api-access-vx422\") pod \"distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:13.112764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:13.112672 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:13.480146 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:13.480124 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr"] Nov 03 22:25:13.482085 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:25:13.482051 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ca0821f_278d_456f_82e9_31578aa0b3a6.slice/crio-8adf3e7530bd2a2fbfd187f8517142eb0602f6ae3d26062050d043ca1c80bc7a WatchSource:0}: Error finding container 8adf3e7530bd2a2fbfd187f8517142eb0602f6ae3d26062050d043ca1c80bc7a: Status 404 returned error can't find the container with id 8adf3e7530bd2a2fbfd187f8517142eb0602f6ae3d26062050d043ca1c80bc7a Nov 03 22:25:13.736939 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:13.736854 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" event={"ID":"4ca0821f-278d-456f-82e9-31578aa0b3a6","Type":"ContainerStarted","Data":"b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0"} Nov 03 22:25:13.736939 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:13.736905 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" event={"ID":"4ca0821f-278d-456f-82e9-31578aa0b3a6","Type":"ContainerStarted","Data":"8adf3e7530bd2a2fbfd187f8517142eb0602f6ae3d26062050d043ca1c80bc7a"} Nov 03 22:25:13.845602 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:13.845555 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podStartSLOduration=1.8455426830000001 podStartE2EDuration="1.845542683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:13.843719342 +0000 UTC m=+1033.255905384" watchObservedRunningTime="2026-11-03 22:25:13.845542683 +0000 UTC m=+1033.257728699" Nov 03 22:25:14.113548 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:14.113458 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:14.450938 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:14.450909 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:25:23.113100 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:23.113066 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:23.125610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:23.125587 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:23.806699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:23.806663 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:25:28.999443 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:25:28.999403 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:25:49.345150 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:49.345113 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:25:56.609463 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.609421 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm"] Nov 03 22:25:56.613446 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.613422 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.665877 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.665845 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm"] Nov 03 22:25:56.709926 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.709888 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"openshift-service-ca.crt\"" Nov 03 22:25:56.710104 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.709960 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"default-dockercfg-rpq9b\"" Nov 03 22:25:56.710104 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.709996 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"kube-root-ca.crt\"" Nov 03 22:25:56.710104 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.710026 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"distinct-w2-906673-kserve-self-signed-certs\"" Nov 03 22:25:56.720258 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.720233 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-home\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.720412 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.720270 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbbd364-d950-4467-85d3-fa4d0740adb8-tls-certs\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.720412 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.720295 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4fnv\" (UniqueName: \"kubernetes.io/projected/8dbbd364-d950-4467-85d3-fa4d0740adb8-kube-api-access-s4fnv\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.720537 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.720415 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-tmp-dir\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.720588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.720570 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-model-cache\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.720626 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.720620 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-dshm\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.821882 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.821845 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-model-cache\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.822084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.821939 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-dshm\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.822084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.822056 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-home\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.822229 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.822114 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbbd364-d950-4467-85d3-fa4d0740adb8-tls-certs\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.822229 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.822166 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4fnv\" (UniqueName: \"kubernetes.io/projected/8dbbd364-d950-4467-85d3-fa4d0740adb8-kube-api-access-s4fnv\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.822229 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.822223 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-tmp-dir\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.822382 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.822283 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-model-cache\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.822437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.822399 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-home\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.822511 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.822483 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-tmp-dir\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.824535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.824486 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-dshm\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.824780 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.824759 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbbd364-d950-4467-85d3-fa4d0740adb8-tls-certs\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.879411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.879332 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4fnv\" (UniqueName: \"kubernetes.io/projected/8dbbd364-d950-4467-85d3-fa4d0740adb8-kube-api-access-s4fnv\") pod \"distinct-w2-906673-kserve-66c87876-kkmzm\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:56.926467 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:56.926435 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:57.294116 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:57.294089 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm"] Nov 03 22:25:57.297219 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:25:57.297190 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dbbd364_d950_4467_85d3_fa4d0740adb8.slice/crio-8830c4bbf8fd9b26ffaa4a353e0cad5d90a296bac2fd218c04d221c15f9d2142 WatchSource:0}: Error finding container 8830c4bbf8fd9b26ffaa4a353e0cad5d90a296bac2fd218c04d221c15f9d2142: Status 404 returned error can't find the container with id 8830c4bbf8fd9b26ffaa4a353e0cad5d90a296bac2fd218c04d221c15f9d2142 Nov 03 22:25:57.996584 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:57.996545 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" event={"ID":"8dbbd364-d950-4467-85d3-fa4d0740adb8","Type":"ContainerStarted","Data":"f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b"} Nov 03 22:25:57.997031 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:57.996598 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" event={"ID":"8dbbd364-d950-4467-85d3-fa4d0740adb8","Type":"ContainerStarted","Data":"8830c4bbf8fd9b26ffaa4a353e0cad5d90a296bac2fd218c04d221c15f9d2142"} Nov 03 22:25:58.104823 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:58.104763 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podStartSLOduration=2.104749363 podStartE2EDuration="2.104749363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:58.103444226 +0000 UTC m=+1077.515630243" watchObservedRunningTime="2026-11-03 22:25:58.104749363 +0000 UTC m=+1077.516935407" Nov 03 22:25:58.926580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:25:58.926552 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:25:58.995996 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:25:58.995949 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:26:06.417173 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.417138 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt"] Nov 03 22:26:06.421429 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.421406 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.476221 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.476186 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt"] Nov 03 22:26:06.519848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.519819 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f7e6db-openshift-default-dockercfg-dfr7z\"" Nov 03 22:26:06.535168 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.535133 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.535339 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.535180 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.535339 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.535294 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.535436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.535372 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.535481 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.535455 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.535579 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.535566 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.535618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.535588 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmnhs\" (UniqueName: \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-kube-api-access-zmnhs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.535661 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.535639 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.535699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.535664 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.636876 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.636832 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637083 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.636885 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637083 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.636952 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637083 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.636979 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnhs\" (UniqueName: \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-kube-api-access-zmnhs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637083 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.637012 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637083 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.637041 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637383 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.637133 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637383 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.637158 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637383 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.637187 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637383 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.637342 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637383 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.637350 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.637426 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.637791 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.637976 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.637846 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.639606 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.639587 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.640045 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.640027 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.691282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.691248 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.691445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.691380 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmnhs\" (UniqueName: \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-kube-api-access-zmnhs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.736720 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.736686 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:06.927014 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.926979 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:26:06.940549 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:06.940448 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:26:07.059848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:07.059757 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:26:07.201520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:07.201476 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt"] Nov 03 22:26:07.204232 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:26:07.203659 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff3dd17_b8e0_4ce1_81d4_8ab4b72e1ee3.slice/crio-199df3ebceb28bfd79b115308a047a037026f6e659b5d3c4e847520df3041aba WatchSource:0}: Error finding container 199df3ebceb28bfd79b115308a047a037026f6e659b5d3c4e847520df3041aba: Status 404 returned error can't find the container with id 199df3ebceb28bfd79b115308a047a037026f6e659b5d3c4e847520df3041aba Nov 03 22:26:07.206447 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:07.206412 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:07.206576 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:07.206539 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:07.206643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:07.206576 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:08.069133 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:08.069094 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" event={"ID":"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3","Type":"ContainerStarted","Data":"65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361"} Nov 03 22:26:08.069542 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:08.069161 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" event={"ID":"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3","Type":"ContainerStarted","Data":"199df3ebceb28bfd79b115308a047a037026f6e659b5d3c4e847520df3041aba"} Nov 03 22:26:08.180299 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:08.180245 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" podStartSLOduration=2.180230878 podStartE2EDuration="2.180230878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:08.177622973 +0000 UTC m=+1087.589808987" watchObservedRunningTime="2026-11-03 22:26:08.180230878 +0000 UTC m=+1087.592416891" Nov 03 22:26:08.737410 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:08.737372 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:08.762521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:08.761369 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:26:08.762521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:08.761442 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 22:26:09.737109 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:09.737066 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:26:09.737558 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:09.737133 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 22:26:10.737186 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:10.737148 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:26:10.737612 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:10.737206 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 22:26:11.737277 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:11.737236 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:26:11.737713 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:11.737301 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 22:26:12.737289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:12.737249 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:26:12.737724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:12.737315 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 22:26:13.737795 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:13.737760 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:26:13.738280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:13.737823 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 22:26:14.741536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:14.741478 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:14.742009 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:14.741807 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:14.742480 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:14.742463 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:26:28.996450 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:26:28.996411 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:26:29.917851 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:29.917816 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:26:49.790961 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.790924 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd"] Nov 03 22:26:49.795080 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.795059 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.846342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.846320 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3378fd-openshift-default-dockercfg-7fv79\"" Nov 03 22:26:49.873470 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.873443 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.873607 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.873479 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.873691 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.873668 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.873802 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.873780 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.873848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.873832 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.873885 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.873863 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10c1781-8bc9-4711-88b6-8b406d29ec25-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.873990 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.873971 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhn8p\" (UniqueName: \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-kube-api-access-lhn8p\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.874056 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.874008 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.874094 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.874061 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.896185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.896158 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd"] Nov 03 22:26:49.898409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.898387 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc"] Nov 03 22:26:49.902305 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.902286 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:49.951291 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.951263 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"distinct-2-w0-f61714-kserve-self-signed-certs\"" Nov 03 22:26:49.975517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.975475 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tls-certs\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:49.975662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.975540 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.975662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.975588 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.975662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.975605 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10c1781-8bc9-4711-88b6-8b406d29ec25-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.975823 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.975790 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-model-cache\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:49.975880 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.975843 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhn8p\" (UniqueName: \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-kube-api-access-lhn8p\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.975941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.975914 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.975941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.975927 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-home\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:49.976051 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.975983 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.976051 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976011 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.976159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976070 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.976159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976103 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.976282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976165 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rr4s\" (UniqueName: \"kubernetes.io/projected/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-kube-api-access-2rr4s\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:49.976282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976220 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tmp-dir\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:49.976389 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976307 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.976389 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976340 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-dshm\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:49.976389 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976343 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.976562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976470 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.976562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976467 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10c1781-8bc9-4711-88b6-8b406d29ec25-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.976713 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.976691 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.978422 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.978401 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.978570 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.978455 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:49.987711 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:49.987682 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc"] Nov 03 22:26:50.029313 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.029275 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:50.029629 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.029606 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhn8p\" (UniqueName: \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-kube-api-access-lhn8p\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:50.077464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.077393 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tls-certs\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.077464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.077442 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-model-cache\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.077705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.077467 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-home\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.077705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.077565 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rr4s\" (UniqueName: \"kubernetes.io/projected/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-kube-api-access-2rr4s\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.077705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.077594 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tmp-dir\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.077705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.077674 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-dshm\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.077951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.077930 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-model-cache\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.078012 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.077990 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-home\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.078057 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.078030 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tmp-dir\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.079745 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.079725 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-dshm\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.079923 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.079907 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tls-certs\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.108731 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.108710 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:50.131282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.131250 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rr4s\" (UniqueName: \"kubernetes.io/projected/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-kube-api-access-2rr4s\") pod \"distinct-2-w0-f61714-kserve-5775949fff-89qvc\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.217891 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.217857 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:50.524998 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.524970 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd"] Nov 03 22:26:50.527283 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:26:50.527249 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10c1781_8bc9_4711_88b6_8b406d29ec25.slice/crio-58dde70996a5bc0e035e90d0c559cdf4e6bea0ad838e33e8560c6acbc9ba5f9c WatchSource:0}: Error finding container 58dde70996a5bc0e035e90d0c559cdf4e6bea0ad838e33e8560c6acbc9ba5f9c: Status 404 returned error can't find the container with id 58dde70996a5bc0e035e90d0c559cdf4e6bea0ad838e33e8560c6acbc9ba5f9c Nov 03 22:26:50.529915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.529882 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:50.530008 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.529947 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:50.530008 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.529976 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:50.619258 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:50.619230 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc"] Nov 03 22:26:50.621848 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:26:50.621820 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ecb248e_7c95_4a4d_9892_57d24a6b1dc2.slice/crio-c5833aee302591dd73866f081659a90895ce0b5341b5eb40431be55fddf07d86 WatchSource:0}: Error finding container c5833aee302591dd73866f081659a90895ce0b5341b5eb40431be55fddf07d86: Status 404 returned error can't find the container with id c5833aee302591dd73866f081659a90895ce0b5341b5eb40431be55fddf07d86 Nov 03 22:26:51.329447 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:51.329407 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" event={"ID":"e10c1781-8bc9-4711-88b6-8b406d29ec25","Type":"ContainerStarted","Data":"e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700"} Nov 03 22:26:51.330015 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:51.329467 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" event={"ID":"e10c1781-8bc9-4711-88b6-8b406d29ec25","Type":"ContainerStarted","Data":"58dde70996a5bc0e035e90d0c559cdf4e6bea0ad838e33e8560c6acbc9ba5f9c"} Nov 03 22:26:51.331259 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:51.331232 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" event={"ID":"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2","Type":"ContainerStarted","Data":"d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361"} Nov 03 22:26:51.331372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:51.331267 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" event={"ID":"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2","Type":"ContainerStarted","Data":"c5833aee302591dd73866f081659a90895ce0b5341b5eb40431be55fddf07d86"} Nov 03 22:26:51.456536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:51.447427 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" podStartSLOduration=2.44740627 podStartE2EDuration="2.44740627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:51.444052924 +0000 UTC m=+1130.856238955" watchObservedRunningTime="2026-11-03 22:26:51.44740627 +0000 UTC m=+1130.859592291" Nov 03 22:26:51.566758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:51.564010 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podStartSLOduration=2.563989827 podStartE2EDuration="2.563989827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:51.561889573 +0000 UTC m=+1130.974075582" watchObservedRunningTime="2026-11-03 22:26:51.563989827 +0000 UTC m=+1130.976175846" Nov 03 22:26:52.109142 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:52.109107 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:52.110685 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:52.110653 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:26:52.110841 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:52.110710 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:26:52.218374 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:52.218343 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:26:53.109512 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:53.109469 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:26:53.111982 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:53.109557 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:26:54.109754 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:54.109718 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:26:54.110233 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:54.109778 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:26:55.110085 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:55.110047 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:26:55.110490 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:55.110105 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:26:56.110036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:56.109983 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:26:56.110246 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:56.110060 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 22:26:57.450004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:57.449965 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:57.450524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:57.450393 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:57.451953 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:26:57.451930 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:26:58.997233 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:26:58.997199 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:27:00.218746 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:00.218714 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:27:00.231922 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:00.231891 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:27:00.425301 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:00.425273 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:27:02.728282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:02.728245 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:27:07.147788 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.147754 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t"] Nov 03 22:27:07.152435 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.152410 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.204027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.203999 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6a6b7f-openshift-default-dockercfg-fqnpk\"" Nov 03 22:27:07.204709 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.204690 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlkzr\" (UniqueName: \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-kube-api-access-rlkzr\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.204775 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.204724 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.204775 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.204750 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.204905 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.204883 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.204966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.204954 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.205068 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.205053 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.205143 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.205126 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.205221 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.205209 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00367384-f2ca-4817-a170-5720fcb0b1e3-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.205279 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.205240 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.253707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.253678 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t"] Nov 03 22:27:07.306111 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306078 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306302 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306139 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306302 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306187 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306302 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306230 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306302 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306251 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00367384-f2ca-4817-a170-5720fcb0b1e3-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306302 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306271 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306328 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlkzr\" (UniqueName: \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-kube-api-access-rlkzr\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306353 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306383 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306590 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306829 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306710 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306829 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306715 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.306829 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.306814 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.307288 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.307260 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00367384-f2ca-4817-a170-5720fcb0b1e3-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.308564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.308539 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.308663 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.308646 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.360128 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.360099 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.360263 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.360240 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlkzr\" (UniqueName: \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-kube-api-access-rlkzr\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.466045 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.466018 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:07.881910 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.881827 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t"] Nov 03 22:27:07.919670 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:27:07.919642 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00367384_f2ca_4817_a170_5720fcb0b1e3.slice/crio-193be087b3b7e2d0ba36d1ed9aa22645b56d4f853e030457dce2592910ddd693 WatchSource:0}: Error finding container 193be087b3b7e2d0ba36d1ed9aa22645b56d4f853e030457dce2592910ddd693: Status 404 returned error can't find the container with id 193be087b3b7e2d0ba36d1ed9aa22645b56d4f853e030457dce2592910ddd693 Nov 03 22:27:07.921909 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.921875 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:07.921996 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.921944 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:07.921996 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:07.921972 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:08.465368 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:08.465331 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" event={"ID":"00367384-f2ca-4817-a170-5720fcb0b1e3","Type":"ContainerStarted","Data":"d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982"} Nov 03 22:27:08.465883 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:08.465392 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" event={"ID":"00367384-f2ca-4817-a170-5720fcb0b1e3","Type":"ContainerStarted","Data":"193be087b3b7e2d0ba36d1ed9aa22645b56d4f853e030457dce2592910ddd693"} Nov 03 22:27:08.466658 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:08.466641 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:08.468332 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:08.468310 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:27:08.468440 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:08.468349 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 22:27:09.466590 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:09.466553 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:27:09.467116 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:09.466610 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 22:27:10.467065 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:10.467022 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:27:10.467481 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:10.467085 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 22:27:11.466478 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:11.466435 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:27:11.466712 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:11.466518 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 22:27:12.466879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:12.466838 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:27:12.467396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:12.466921 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 22:27:13.467051 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:13.467011 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:27:13.467450 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:13.467069 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 22:27:14.467318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:14.467280 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:27:14.467766 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:14.467385 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 22:27:15.471789 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:15.471760 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:15.472177 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:15.472159 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:15.472951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:15.472935 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:27:15.629131 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:15.629074 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" podStartSLOduration=8.629061849 podStartE2EDuration="8.629061849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:08.575708343 +0000 UTC m=+1147.987894359" watchObservedRunningTime="2026-11-03 22:27:15.629061849 +0000 UTC m=+1155.041247866" Nov 03 22:27:29.002223 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:27:29.002174 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:27:47.058695 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:27:47.058663 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:27:58.998484 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:27:58.998448 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:28:19.277599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.277564 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd"] Nov 03 22:28:19.281465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.281443 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.335455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.335420 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd"] Nov 03 22:28:19.373084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.373051 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-model-cache\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.373232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.373180 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-tmp-dir\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.373278 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.373251 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b4fadd-1467-4cab-991b-08a56498eeae-tls-certs\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.373385 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.373369 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jxtm\" (UniqueName: \"kubernetes.io/projected/31b4fadd-1467-4cab-991b-08a56498eeae-kube-api-access-8jxtm\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.373440 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.373426 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-dshm\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.373563 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.373541 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-home\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.379020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.378999 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-906673\"/\"distinct-2-w2-906673-kserve-self-signed-certs\"" Nov 03 22:28:19.474643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.474602 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b4fadd-1467-4cab-991b-08a56498eeae-tls-certs\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.474866 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.474728 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jxtm\" (UniqueName: \"kubernetes.io/projected/31b4fadd-1467-4cab-991b-08a56498eeae-kube-api-access-8jxtm\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.474866 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.474799 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-dshm\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.474866 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.474856 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-home\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.475048 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.474905 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-model-cache\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.475048 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.474992 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-tmp-dir\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.475479 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.475414 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-home\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.475479 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.475444 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-model-cache\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.475681 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.475524 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-tmp-dir\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.477262 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.477242 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-dshm\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.477355 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.477296 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b4fadd-1467-4cab-991b-08a56498eeae-tls-certs\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.528866 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.528798 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jxtm\" (UniqueName: \"kubernetes.io/projected/31b4fadd-1467-4cab-991b-08a56498eeae-kube-api-access-8jxtm\") pod \"distinct-2-w2-906673-kserve-85665659c5-6gdvd\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.594562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.594529 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:19.965327 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:19.965297 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd"] Nov 03 22:28:19.970098 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:28:19.970070 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31b4fadd_1467_4cab_991b_08a56498eeae.slice/crio-4d7e2abde2c2b9223958af157915b727f3087ca81e9e3a406f260be3328e342b WatchSource:0}: Error finding container 4d7e2abde2c2b9223958af157915b727f3087ca81e9e3a406f260be3328e342b: Status 404 returned error can't find the container with id 4d7e2abde2c2b9223958af157915b727f3087ca81e9e3a406f260be3328e342b Nov 03 22:28:20.904946 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:20.904905 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" event={"ID":"31b4fadd-1467-4cab-991b-08a56498eeae","Type":"ContainerStarted","Data":"00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91"} Nov 03 22:28:20.905427 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:20.904958 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" event={"ID":"31b4fadd-1467-4cab-991b-08a56498eeae","Type":"ContainerStarted","Data":"4d7e2abde2c2b9223958af157915b727f3087ca81e9e3a406f260be3328e342b"} Nov 03 22:28:21.014164 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:21.014114 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podStartSLOduration=2.014101298 podStartE2EDuration="2.014101298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:21.011988687 +0000 UTC m=+1220.424174705" watchObservedRunningTime="2026-11-03 22:28:21.014101298 +0000 UTC m=+1220.426287314" Nov 03 22:28:21.594698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:21.594667 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:22.116505 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.116471 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz"] Nov 03 22:28:22.121307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.121282 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.206921 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.206887 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f18706-2830-4b62-af04-df1bc307ecda-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.207118 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.206942 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f18706-2830-4b62-af04-df1bc307ecda-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.207118 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.207043 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.207118 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.207090 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.207270 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.207127 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.207351 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.207330 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.207471 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.207413 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w4sq\" (UniqueName: \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-kube-api-access-8w4sq\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.207556 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.207488 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.207556 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.207544 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.220363 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.220333 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz"] Nov 03 22:28:22.308575 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.308539 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.308734 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.308658 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.308734 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.308707 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w4sq\" (UniqueName: \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-kube-api-access-8w4sq\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.308849 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.308822 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.308904 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.308877 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.308942 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.308929 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f18706-2830-4b62-af04-df1bc307ecda-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.308991 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.308937 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.309046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.308989 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f18706-2830-4b62-af04-df1bc307ecda-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.309046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.309021 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.309046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.309032 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.309200 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.309071 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.309200 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.309179 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.309313 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.309289 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.309690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.309670 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f18706-2830-4b62-af04-df1bc307ecda-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.311247 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.311222 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.311406 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.311391 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f18706-2830-4b62-af04-df1bc307ecda-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.362811 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.362779 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.362940 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.362908 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w4sq\" (UniqueName: \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-kube-api-access-8w4sq\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.435549 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.435523 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:22.858084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.858007 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz"] Nov 03 22:28:22.862360 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:28:22.862333 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f18706_2830_4b62_af04_df1bc307ecda.slice/crio-20ca115a25c571cce9009899f72ecf5ed900a57e7812b42e95fc7ff720f6c298 WatchSource:0}: Error finding container 20ca115a25c571cce9009899f72ecf5ed900a57e7812b42e95fc7ff720f6c298: Status 404 returned error can't find the container with id 20ca115a25c571cce9009899f72ecf5ed900a57e7812b42e95fc7ff720f6c298 Nov 03 22:28:22.864761 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.864733 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:22.864834 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.864794 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:22.864834 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.864824 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:22.919066 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:22.919033 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" event={"ID":"80f18706-2830-4b62-af04-df1bc307ecda","Type":"ContainerStarted","Data":"20ca115a25c571cce9009899f72ecf5ed900a57e7812b42e95fc7ff720f6c298"} Nov 03 22:28:23.926942 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:23.926905 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" event={"ID":"80f18706-2830-4b62-af04-df1bc307ecda","Type":"ContainerStarted","Data":"dab454577d84c14de1f49bcb15ce8a8df95d917d48aabae1f823df4a718b83ad"} Nov 03 22:28:24.085238 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:24.085185 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" podStartSLOduration=2.085170614 podStartE2EDuration="2.085170614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:24.082473927 +0000 UTC m=+1223.494659927" watchObservedRunningTime="2026-11-03 22:28:24.085170614 +0000 UTC m=+1223.497356630" Nov 03 22:28:24.436679 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:24.436642 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:24.438514 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:24.438465 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:28:24.438667 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:24.438557 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:28:25.436467 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:25.436436 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:28:25.436885 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:25.436484 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:28:26.436864 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:26.436833 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:28:26.437269 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:26.436883 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:28:27.436639 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:27.436601 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:28:27.436816 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:27.436666 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:28:28.436085 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:28.436050 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:28:28.436469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:28.436107 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:28:28.998439 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:28:28.998393 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:28:29.436607 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:29.436577 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:28:29.436970 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:29.436655 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 22:28:29.595234 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:29.595201 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:29.608159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:29.608133 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:30.001770 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:30.001737 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:28:30.440289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:30.440254 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:30.440700 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:30.440521 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:30.441342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:30.441324 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:28:37.873959 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:28:37.873923 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:28:58.996266 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:28:58.996230 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:29:06.243671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.243631 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh"] Nov 03 22:29:06.249203 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.249175 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.307689 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.307655 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh"] Nov 03 22:29:06.362733 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.362697 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.362997 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.362974 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbd7w\" (UniqueName: \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-kube-api-access-gbd7w\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.363060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.363028 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.363121 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.363098 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.363178 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.363161 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.363331 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.363310 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.363377 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.363346 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.363464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.363445 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.363566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.363543 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.465241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.465191 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.465430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.465274 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.465430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.465390 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.465584 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.465473 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbd7w\" (UniqueName: \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-kube-api-access-gbd7w\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.465584 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.465552 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.465698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.465583 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.465698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.465632 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.465799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.465704 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.465799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.465728 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.466289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.466253 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.466478 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.466455 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.466635 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.466549 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.466766 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.466650 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.467282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.467257 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.468124 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.468095 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.468423 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.468403 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.518955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.518882 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.519210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.519178 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbd7w\" (UniqueName: \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-kube-api-access-gbd7w\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.562705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.562673 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:06.761921 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.761882 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv"] Nov 03 22:29:06.767183 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.767159 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.818260 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.818231 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f61714-openshift-default-dockercfg-wvcgf\"" Nov 03 22:29:06.821535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.821469 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv"] Nov 03 22:29:06.870282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.870253 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f2da67-217f-4b4a-b606-deb7afc23093-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.870443 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.870357 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.870443 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.870406 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.870443 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.870427 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m44sd\" (UniqueName: \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-kube-api-access-m44sd\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.870600 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.870584 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.870652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.870642 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.870771 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.870750 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f2da67-217f-4b4a-b606-deb7afc23093-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.870858 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.870842 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.870906 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.870890 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972058 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972278 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972100 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f2da67-217f-4b4a-b606-deb7afc23093-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972278 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972166 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972278 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972192 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972278 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972236 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f2da67-217f-4b4a-b606-deb7afc23093-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972293 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972360 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972394 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m44sd\" (UniqueName: \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-kube-api-access-m44sd\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972483 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972525 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972579 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972738 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.972950 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.972929 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.973363 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.973340 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f2da67-217f-4b4a-b606-deb7afc23093-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.974647 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.974627 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.974800 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.974782 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f2da67-217f-4b4a-b606-deb7afc23093-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:06.989035 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.989013 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh"] Nov 03 22:29:06.994109 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:29:06.994082 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d426a10_fbce_4d15_bb78_cb3d3b40e811.slice/crio-6f8480b4b04d2c561b16a288afb33ade4cca7480764b76c6f76ef618236ba3c3 WatchSource:0}: Error finding container 6f8480b4b04d2c561b16a288afb33ade4cca7480764b76c6f76ef618236ba3c3: Status 404 returned error can't find the container with id 6f8480b4b04d2c561b16a288afb33ade4cca7480764b76c6f76ef618236ba3c3 Nov 03 22:29:06.996674 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.996645 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:06.996756 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.996726 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:06.996756 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:06.996753 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:07.027414 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.027357 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:07.027414 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.027383 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m44sd\" (UniqueName: \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-kube-api-access-m44sd\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:07.081307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.081270 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:07.084586 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.084560 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d"] Nov 03 22:29:07.094793 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.094762 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.144478 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.144440 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-906673-openshift-default-dockercfg-cwtcb\"" Nov 03 22:29:07.151873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.151830 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d"] Nov 03 22:29:07.178724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.177954 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.178724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.178010 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.178724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.178070 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.178724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.178100 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6861036d-82cd-413b-88c2-df43b2b7a613-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.178724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.178304 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.178724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.178334 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6861036d-82cd-413b-88c2-df43b2b7a613-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.178724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.178359 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.178724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.178416 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sz9j\" (UniqueName: \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-kube-api-access-7sz9j\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.178724 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.178446 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.213802 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.213764 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" event={"ID":"6d426a10-fbce-4d15-bb78-cb3d3b40e811","Type":"ContainerStarted","Data":"5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11"} Nov 03 22:29:07.215142 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.215115 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" event={"ID":"6d426a10-fbce-4d15-bb78-cb3d3b40e811","Type":"ContainerStarted","Data":"6f8480b4b04d2c561b16a288afb33ade4cca7480764b76c6f76ef618236ba3c3"} Nov 03 22:29:07.279269 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279177 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sz9j\" (UniqueName: \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-kube-api-access-7sz9j\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.279269 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279212 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.279825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279291 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.279825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279339 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.279825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279392 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.279825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279418 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6861036d-82cd-413b-88c2-df43b2b7a613-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.279825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279639 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.279825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279674 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6861036d-82cd-413b-88c2-df43b2b7a613-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.279825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279704 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.279825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279729 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.279825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.279789 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.280301 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.280057 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.280301 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.280230 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.280685 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.280659 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6861036d-82cd-413b-88c2-df43b2b7a613-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.281961 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.281940 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.282099 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.282071 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6861036d-82cd-413b-88c2-df43b2b7a613-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.327047 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.327002 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podStartSLOduration=1.326989758 podStartE2EDuration="1.326989758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:07.325517668 +0000 UTC m=+1266.737703682" watchObservedRunningTime="2026-11-03 22:29:07.326989758 +0000 UTC m=+1266.739175774" Nov 03 22:29:07.336414 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.336379 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sz9j\" (UniqueName: \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-kube-api-access-7sz9j\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.336721 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.336701 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.412362 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.412322 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:07.479751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.477947 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv"] Nov 03 22:29:07.492758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.490320 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:07.492758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.490808 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:07.492758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.491117 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:07.562779 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.562751 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:07.816881 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.816782 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d"] Nov 03 22:29:07.822159 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:29:07.822126 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6861036d_82cd_413b_88c2_df43b2b7a613.slice/crio-df21c10b39562fe6c9c92aed7ca1afef2f0ccade38d4d10b38ae59fd0828079f WatchSource:0}: Error finding container df21c10b39562fe6c9c92aed7ca1afef2f0ccade38d4d10b38ae59fd0828079f: Status 404 returned error can't find the container with id df21c10b39562fe6c9c92aed7ca1afef2f0ccade38d4d10b38ae59fd0828079f Nov 03 22:29:07.825068 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.825035 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:07.825175 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.825107 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:07.825175 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:07.825137 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:08.123123 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.122735 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68"] Nov 03 22:29:08.127723 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.127691 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.196217 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.196170 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68"] Nov 03 22:29:08.208272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.206536 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.208272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.206667 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7zg\" (UniqueName: \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-kube-api-access-gt7zg\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.208272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.206801 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.208272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.206833 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.208272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.206894 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.208272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.206935 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.208272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.207052 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.208272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.207078 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.208272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.207143 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.245918 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.245881 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" event={"ID":"6861036d-82cd-413b-88c2-df43b2b7a613","Type":"ContainerStarted","Data":"bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8"} Nov 03 22:29:08.246166 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.246149 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" event={"ID":"6861036d-82cd-413b-88c2-df43b2b7a613","Type":"ContainerStarted","Data":"df21c10b39562fe6c9c92aed7ca1afef2f0ccade38d4d10b38ae59fd0828079f"} Nov 03 22:29:08.250451 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.250420 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" event={"ID":"11f2da67-217f-4b4a-b606-deb7afc23093","Type":"ContainerStarted","Data":"af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0"} Nov 03 22:29:08.250648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.250631 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" event={"ID":"11f2da67-217f-4b4a-b606-deb7afc23093","Type":"ContainerStarted","Data":"cefefb3cfabe4498477745badf53eedff2590f84238d02b1cc81ff7495c9fc14"} Nov 03 22:29:08.322393 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.322344 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.329225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.326587 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.329225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.326645 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.329225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.326750 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.329225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.326781 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.329225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.326821 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7zg\" (UniqueName: \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-kube-api-access-gt7zg\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.329225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.327081 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.329225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.327130 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.329225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.327224 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.329225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.327274 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.329225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.328361 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.330742 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.330715 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.331401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.331378 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.332363 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.332044 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.332363 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.332129 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.334720 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.334698 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.401320 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.401242 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7zg\" (UniqueName: \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-kube-api-access-gt7zg\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.406857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.406820 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.418290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.416350 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:08.418290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.416823 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:29:08.418290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.416884 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:29:08.480189 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.479758 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:08.577545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.566858 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podStartSLOduration=2.566837959 podStartE2EDuration="2.566837959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:08.404012502 +0000 UTC m=+1267.816198521" watchObservedRunningTime="2026-11-03 22:29:08.566837959 +0000 UTC m=+1267.979023977" Nov 03 22:29:08.583113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.583075 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:29:08.583396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:08.583366 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 22:29:09.025559 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.024339 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podStartSLOduration=2.024320635 podStartE2EDuration="2.024320635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:08.58653232 +0000 UTC m=+1267.998718375" watchObservedRunningTime="2026-11-03 22:29:09.024320635 +0000 UTC m=+1268.436506654" Nov 03 22:29:09.061971 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.036534 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68"] Nov 03 22:29:09.061971 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.037359 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:09.061971 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.037432 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:09.061971 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.037477 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:09.096744 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.095409 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:09.096744 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.095557 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:29:09.096744 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.095600 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:29:09.345532 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.337417 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" event={"ID":"d24285a4-1ad4-4d6b-96da-75f575d24b5e","Type":"ContainerStarted","Data":"4c93a5981cc7813b7124d2e46a114416a52aa48157101bc5e8ab62b6754a1126"} Nov 03 22:29:09.422894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.412789 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:29:09.422894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.412847 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:29:09.572524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.572392 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:29:09.572524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:09.572441 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 22:29:10.045518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.044640 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.045518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.045045 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 22:29:10.061519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.059649 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.083520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.081910 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:29:10.083520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.081973 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:29:10.108519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.107586 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.184519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.177594 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.274519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.268011 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.288518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.284870 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 22:29:10.382349 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.382247 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.416739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.415360 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:29:10.416739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.415633 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:29:10.450517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.450125 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 22:29:10.505527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.496821 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" event={"ID":"d24285a4-1ad4-4d6b-96da-75f575d24b5e","Type":"ContainerStarted","Data":"2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69"} Nov 03 22:29:10.565530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.563882 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:29:10.565530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.563941 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 22:29:10.616554 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.613781 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podStartSLOduration=2.6137616120000002 podStartE2EDuration="2.613761612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:10.609330837 +0000 UTC m=+1270.021516866" watchObservedRunningTime="2026-11-03 22:29:10.613761612 +0000 UTC m=+1270.025947632" Nov 03 22:29:10.616554 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.614014 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Nov 03 22:29:10.653578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.651620 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.760039 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.759994 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.910581 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:10.907297 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 03 22:29:11.059526 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.058822 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.059526 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.059230 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 22:29:11.063517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.059904 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:29:11.063517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.059968 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:29:11.088518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.082817 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.088518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.083247 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:29:11.088518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.084841 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:29:11.088518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.084879 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:29:11.203520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.197184 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.203520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.197601 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:29:11.248519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.237316 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.369521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.368659 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.369521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.369075 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 03 22:29:11.369521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.369200 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:29:11.409520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.370760 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.409520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.370921 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:29:11.410298 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.410196 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.413486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.413385 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:29:11.413486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.413433 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:29:11.485578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.464653 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 22:29:11.485578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.465120 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:29:11.485578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.467456 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 22:29:11.514949 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.512562 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:29:11.514949 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.512618 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:11.516947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.516826 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:29:11.516947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.516881 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 22:29:11.587665 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.587615 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.652253 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.652215 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 22:29:11.653588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.653565 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:29:11.720486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.720452 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:29:11.745785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.745595 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 22:29:11.765545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.750587 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.767353 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.767314 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.767997 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.767977 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:29:11.810535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.807282 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 22:29:11.942520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:11.941588 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.025419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.007137 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:29:12.073517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.063616 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.073517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.064906 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.098520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.097830 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.098520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.097880 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:29:12.098784 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.098564 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.205523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.204614 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.386579 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.382780 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.417470 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.383626 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.480529 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.469202 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 22:29:12.484770 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.484732 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.485029 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.484994 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:29:12.485794 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.485755 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.486020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.485995 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 22:29:12.486898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.486872 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.623324 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.623189 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.722912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.691787 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 22:29:12.761598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.752050 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.761598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.752163 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.784018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.773621 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.817013 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.812693 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 22:29:12.885672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.885624 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.989604 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:12.970824 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 03 22:29:13.066543 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.064199 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 22:29:13.071770 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.071725 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.205519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.079122 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:29:13.227575 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.224716 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.234212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.232833 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 22:29:13.234212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.142611 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:29:13.234212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.233085 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:29:13.234212 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.101349 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.253275 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.253175 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.414550 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.407999 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:29:13.414550 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.408090 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:29:13.478554 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.433660 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.478554 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.436240 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 22:29:13.478554 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.442148 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:29:13.478554 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.442204 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:29:13.510273 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.510167 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 22:29:13.578533 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.540091 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.805043 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.566398 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:29:13.805296 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.805255 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 22:29:13.805732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.776764 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.806162 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.777942 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.836188 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.774940 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 22:29:13.836993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.836432 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 22:29:14.000522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:13.982614 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.086250 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.086144 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.201725 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.201677 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:29:14.202010 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.201984 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:29:14.263862 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.202143 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.264170 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.203765 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.272759 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.227332 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.273002 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.251534 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.320520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.305653 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:29:14.320520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.305728 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 22:29:14.451917 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.417126 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:29:14.451917 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.417182 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:29:14.451917 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.449678 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.451917 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.450061 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 22:29:14.517694 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.517643 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 22:29:14.609520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.605921 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:29:14.609520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.605977 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 22:29:14.621522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.619648 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:29:14.621522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.619704 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 22:29:14.716482 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.716378 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:29:14.816528 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.815582 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.816528 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.815966 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.816528 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.816221 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 03 22:29:14.872525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:14.863690 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 22:29:15.053039 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.052628 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.098521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.096102 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.259565 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.177197 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:29:15.269775 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.269731 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:29:15.270018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.206415 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.307519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.276570 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 22:29:15.335660 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.335615 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.420513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.417572 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.420513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.418234 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:29:15.420513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.418263 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:29:15.450926 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.450882 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.457523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.451425 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.466901 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.458918 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:29:15.499568 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.489045 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:29:15.499568 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.489122 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 22:29:15.520217 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.520176 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:15.570023 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.569935 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 22:29:15.570290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.570259 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 22:29:15.714534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.714071 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:29:15.731539 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.727368 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:29:15.816630 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.816587 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 22:29:15.821572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:15.817195 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:16.099575 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.085658 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:29:16.099575 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.085729 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 22:29:16.432833 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.432796 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:16.464176 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.443832 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:29:16.464176 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.443894 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:29:16.476894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.476832 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:16.511610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.510959 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 22:29:16.565132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.565089 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:29:16.565815 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.565781 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 22:29:16.647966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.647917 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:16.683149 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.683039 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:16.714843 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.698682 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 22:29:16.715392 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.714197 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:16.842563 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.827640 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 22:29:16.842563 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.838565 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:16.869560 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.851351 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:29:16.902520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.895866 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:16.940122 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:16.939207 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:17.068527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.052592 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:29:17.123170 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.090588 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:29:17.130446 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.130414 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:29:17.167525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.165617 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:29:17.359581 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.350777 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:29:17.377530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.366208 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:29:17.439973 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.439936 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:29:17.440241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.440215 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 22:29:17.535089 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.534775 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:29:17.542751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.542715 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:29:17.545029 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.544999 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 22:29:17.607372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.607336 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:29:17.747405 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.747106 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:17.747405 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:17.747153 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:18.122665 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:18.122577 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:18.127701 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:18.127649 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:18.482669 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:18.482633 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:29:18.482947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:18.482917 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 22:29:18.821744 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:18.821640 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:18.822362 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:18.822317 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:18.997940 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:18.997906 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:19.020520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.005165 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:19.023628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.023598 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:29:19.026897 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.026866 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:19.028217 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.028136 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:19.029660 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.029632 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:29:19.128713 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.128622 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:19.128713 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.128692 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:19.456880 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.456852 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:19.457673 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.457653 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:19.459670 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.459645 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:29:19.605709 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.605680 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:19.606432 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.606408 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:19.607957 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:19.607940 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:29:21.931325 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.931290 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8"] Nov 03 22:29:21.935640 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.935617 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:21.981349 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.981314 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:21.981521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.981438 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:21.981597 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.981553 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:21.981641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.981626 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvz6\" (UniqueName: \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-kube-api-access-5zvz6\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:21.981778 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.981754 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57101afc-e8db-439a-b8fe-ba005d025ea3-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:21.981948 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.981927 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:21.982033 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.981969 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:21.982090 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.982081 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:21.982145 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:21.982122 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.031107 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.031073 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8"] Nov 03 22:29:22.083702 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.083660 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.083972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.083732 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvz6\" (UniqueName: \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-kube-api-access-5zvz6\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.083972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.083759 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57101afc-e8db-439a-b8fe-ba005d025ea3-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.083972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.083802 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.083972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.083823 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.083972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.083853 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.083972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.083884 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.083972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.083946 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.084342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.084015 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.084342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.084187 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.084342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.084305 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.084467 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.084344 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.084553 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.084465 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.084604 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.084588 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57101afc-e8db-439a-b8fe-ba005d025ea3-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.086224 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.086201 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.086479 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.086458 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.138915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.138876 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.139069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.138938 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvz6\" (UniqueName: \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-kube-api-access-5zvz6\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.250305 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.250279 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:22.678885 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.678811 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8"] Nov 03 22:29:22.684368 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:29:22.684341 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57101afc_e8db_439a_b8fe_ba005d025ea3.slice/crio-b94197d6032913fd9b4be9d4ef4d796cc95a764e99f15dadcac02c28adc832c9 WatchSource:0}: Error finding container b94197d6032913fd9b4be9d4ef4d796cc95a764e99f15dadcac02c28adc832c9: Status 404 returned error can't find the container with id b94197d6032913fd9b4be9d4ef4d796cc95a764e99f15dadcac02c28adc832c9 Nov 03 22:29:22.687011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.686982 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:22.687095 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.687044 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:22.687095 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.687074 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:22.998509 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.998459 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" event={"ID":"57101afc-e8db-439a-b8fe-ba005d025ea3","Type":"ContainerStarted","Data":"bc8f2571f3586cfe9cf1ce7f9400e3cedf9bf6127b6d372f1f36e05463acb24f"} Nov 03 22:29:22.998957 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:22.998544 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" event={"ID":"57101afc-e8db-439a-b8fe-ba005d025ea3","Type":"ContainerStarted","Data":"b94197d6032913fd9b4be9d4ef4d796cc95a764e99f15dadcac02c28adc832c9"} Nov 03 22:29:23.109891 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:23.109839 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" podStartSLOduration=2.10982545 podStartE2EDuration="2.10982545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:23.109050218 +0000 UTC m=+1282.521236246" watchObservedRunningTime="2026-11-03 22:29:23.10982545 +0000 UTC m=+1282.522011466" Nov 03 22:29:23.259579 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:23.258871 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:23.264374 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:23.264340 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:29:23.264602 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:23.264402 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:29:24.263646 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:24.263608 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:29:24.264158 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:24.263669 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:29:25.263936 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:25.263894 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:29:25.264336 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:25.263964 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:29:26.259521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:26.254880 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:29:26.259521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:26.254946 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:29:27.264624 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:27.257903 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:29:27.264624 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:27.257967 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:29:28.256520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:28.250675 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:29:28.256520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:28.250736 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:29:28.999536 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:29:28.999465 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:29:29.251152 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:29.251067 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:29:29.251152 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:29.251124 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 22:29:30.255699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:30.255669 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:30.256100 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:30.255925 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:30.256628 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:30.256612 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:29:33.435029 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:33.434997 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:29:35.882355 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:35.881625 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:35.990577 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:35.990072 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.074521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.073580 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.180576 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.180412 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx"] Nov 03 22:29:36.242772 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.242719 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.350520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.243011 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 22:29:36.399549 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.358920 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.460519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.457938 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.460519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.458143 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 22:29:36.517520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.509575 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx"] Nov 03 22:29:36.517520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.509735 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.736578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.736081 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.772910 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.824615 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.824733 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.824771 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252da87c-dca5-447a-b98f-ad65e61b8337-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.824801 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252da87c-dca5-447a-b98f-ad65e61b8337-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.824872 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsq2k\" (UniqueName: \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-kube-api-access-wsq2k\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.824934 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.824996 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.825021 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.789388 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.821936 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 03 22:29:36.873580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.840560 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 22:29:36.899519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.885583 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.942307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.932115 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.942307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.932193 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.942307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.932277 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.942307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.932314 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252da87c-dca5-447a-b98f-ad65e61b8337-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.942307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.932342 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252da87c-dca5-447a-b98f-ad65e61b8337-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.942307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.932406 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsq2k\" (UniqueName: \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-kube-api-access-wsq2k\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.942307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.932466 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.942307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.932576 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:36.942307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:36.932606 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:37.069575 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.012864 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 03 22:29:37.151018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.077574 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 22:29:37.151486 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.096076 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:37.181651 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.180451 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:37.181651 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.181584 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:37.182159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.182136 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:37.192532 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.183067 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:29:37.192532 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.183126 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:29:37.192532 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.183165 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:29:37.326567 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.321263 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:29:37.326567 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.322040 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252da87c-dca5-447a-b98f-ad65e61b8337-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:37.338517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.337538 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:37.479534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.471649 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:37.479534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.472944 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:29:37.524592 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.524120 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252da87c-dca5-447a-b98f-ad65e61b8337-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:37.535443 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.535300 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:29:37.546517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.536274 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsq2k\" (UniqueName: \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-kube-api-access-wsq2k\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:37.546517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.537114 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:37.546517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.541069 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:29:37.616081 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.616004 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:29:37.683093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:37.683057 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:38.147160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.147122 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx"] Nov 03 22:29:38.152516 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:29:38.152474 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252da87c_dca5_447a_b98f_ad65e61b8337.slice/crio-ddf0fa526726dabbbbd004e020b5e1fce147617ecc09bb1e3222e347d38e0dcd WatchSource:0}: Error finding container ddf0fa526726dabbbbd004e020b5e1fce147617ecc09bb1e3222e347d38e0dcd: Status 404 returned error can't find the container with id ddf0fa526726dabbbbd004e020b5e1fce147617ecc09bb1e3222e347d38e0dcd Nov 03 22:29:38.155164 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.155132 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:38.155251 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.155203 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:38.155295 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.155258 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:38.383010 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.382980 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm"] Nov 03 22:29:38.387355 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.387333 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.428800 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.428714 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.428800 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.428790 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.429027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.428897 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.429027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.428936 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.429027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.429010 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.429193 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.429053 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.429193 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.429127 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.429303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.429242 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.429303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.429273 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx4dm\" (UniqueName: \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-kube-api-access-gx4dm\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.445146 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.445112 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm"] Nov 03 22:29:38.530143 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530110 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530191 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530217 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx4dm\" (UniqueName: \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-kube-api-access-gx4dm\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530244 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530324 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530397 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530446 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530539 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530548 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530599 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530616 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.530901 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.530842 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.531218 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.531196 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.531312 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.531196 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.533089 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.533070 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.533329 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.533307 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.584907 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.584876 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx4dm\" (UniqueName: \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-kube-api-access-gx4dm\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.585054 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.584919 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.706063 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.706036 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:38.729917 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.729879 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" event={"ID":"252da87c-dca5-447a-b98f-ad65e61b8337","Type":"ContainerStarted","Data":"8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e"} Nov 03 22:29:38.730083 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.729950 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" event={"ID":"252da87c-dca5-447a-b98f-ad65e61b8337","Type":"ContainerStarted","Data":"ddf0fa526726dabbbbd004e020b5e1fce147617ecc09bb1e3222e347d38e0dcd"} Nov 03 22:29:38.880808 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:38.876328 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" podStartSLOduration=2.876292436 podStartE2EDuration="2.876292436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:38.871193442 +0000 UTC m=+1298.283379459" watchObservedRunningTime="2026-11-03 22:29:38.876292436 +0000 UTC m=+1298.288478454" Nov 03 22:29:39.170656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:39.170624 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm"] Nov 03 22:29:39.173916 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:29:39.173888 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691ef725_0a35_48a2_a393_56ddf5b1b3f6.slice/crio-0d0f9938f407b4161a9559f99572a3cbd7f197adf245b982e753917c60cc4169 WatchSource:0}: Error finding container 0d0f9938f407b4161a9559f99572a3cbd7f197adf245b982e753917c60cc4169: Status 404 returned error can't find the container with id 0d0f9938f407b4161a9559f99572a3cbd7f197adf245b982e753917c60cc4169 Nov 03 22:29:39.177027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:39.176991 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:39.177120 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:39.177073 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:39.177195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:39.177117 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:39.684020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:39.683986 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:39.685634 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:39.685607 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:29:39.685760 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:39.685653 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 22:29:39.763612 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:39.763565 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" event={"ID":"691ef725-0a35-48a2-a393-56ddf5b1b3f6","Type":"ContainerStarted","Data":"3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb"} Nov 03 22:29:39.763801 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:39.763635 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" event={"ID":"691ef725-0a35-48a2-a393-56ddf5b1b3f6","Type":"ContainerStarted","Data":"0d0f9938f407b4161a9559f99572a3cbd7f197adf245b982e753917c60cc4169"} Nov 03 22:29:39.880297 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:39.880174 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" podStartSLOduration=1.880154278 podStartE2EDuration="1.880154278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:39.875118293 +0000 UTC m=+1299.287304314" watchObservedRunningTime="2026-11-03 22:29:39.880154278 +0000 UTC m=+1299.292340296" Nov 03 22:29:40.691989 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:40.691910 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:29:40.692607 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:40.692570 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 22:29:40.707712 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:40.707684 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:29:40.707914 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:40.707732 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:29:40.710718 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:40.710695 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:41.687518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:41.686920 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:29:41.687518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:41.686985 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 22:29:41.706725 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:41.706693 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:29:41.707096 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:41.706738 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:29:42.691598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:42.691114 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:29:42.691598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:42.691172 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 22:29:42.707030 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:42.707002 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:29:42.707392 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:42.707045 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:29:43.687523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:43.685880 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:29:43.687523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:43.685942 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 22:29:43.707000 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:43.706976 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:29:43.707195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:43.707178 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:29:44.684517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:44.683922 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:29:44.684517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:44.683998 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 22:29:44.707002 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:44.706967 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:29:44.707175 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:44.707027 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:29:45.707300 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:45.707260 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:29:45.707772 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:45.707344 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 22:29:45.771638 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:45.771026 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:45.772403 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:45.772377 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:45.774480 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:45.774431 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:29:46.999449 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:46.999418 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:46.999941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:46.999916 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:47.000570 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:47.000543 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:29:51.272144 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.272111 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv"] Nov 03 22:29:51.276340 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.276321 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.399834 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.399789 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.400031 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.399871 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.400031 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.399896 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41042da-424e-4520-8234-b11bd45a754d-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.400164 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.400064 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.400225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.400207 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.400276 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.400244 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41042da-424e-4520-8234-b11bd45a754d-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.400330 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.400296 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.400389 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.400351 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjdkj\" (UniqueName: \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-kube-api-access-fjdkj\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.400475 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.400453 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.471616 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.471573 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv"] Nov 03 22:29:51.501838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.501801 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.502041 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.501940 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.502041 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.501981 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41042da-424e-4520-8234-b11bd45a754d-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.502041 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.502012 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.502228 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.502088 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjdkj\" (UniqueName: \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-kube-api-access-fjdkj\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.502228 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.502152 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.502838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.502805 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.502968 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.502314 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.503092 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.503040 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.503213 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.503155 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.503290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.503220 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.503290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.503246 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41042da-424e-4520-8234-b11bd45a754d-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.504081 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.504051 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41042da-424e-4520-8234-b11bd45a754d-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.504233 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.504058 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.506123 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.506098 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41042da-424e-4520-8234-b11bd45a754d-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.506234 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.506117 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.561525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.561417 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.563296 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.563262 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjdkj\" (UniqueName: \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-kube-api-access-fjdkj\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:51.591389 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:51.591363 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:52.024301 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:52.024260 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv"] Nov 03 22:29:52.028809 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:29:52.028782 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41042da_424e_4520_8234_b11bd45a754d.slice/crio-bd4867a5aae22a93acc77126f76a3af7e91a2946a428199da05651aa054f390c WatchSource:0}: Error finding container bd4867a5aae22a93acc77126f76a3af7e91a2946a428199da05651aa054f390c: Status 404 returned error can't find the container with id bd4867a5aae22a93acc77126f76a3af7e91a2946a428199da05651aa054f390c Nov 03 22:29:52.031242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:52.031210 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:52.031318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:52.031273 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:52.031318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:52.031303 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:52.894476 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:52.894440 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" event={"ID":"a41042da-424e-4520-8234-b11bd45a754d","Type":"ContainerStarted","Data":"96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557"} Nov 03 22:29:52.894879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:52.894519 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" event={"ID":"a41042da-424e-4520-8234-b11bd45a754d","Type":"ContainerStarted","Data":"bd4867a5aae22a93acc77126f76a3af7e91a2946a428199da05651aa054f390c"} Nov 03 22:29:53.591560 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:53.591527 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:53.592787 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:53.592759 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:29:53.592911 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:53.592816 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:29:54.592129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:54.592094 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:29:54.592543 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:54.592158 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:29:55.592198 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:55.592167 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:29:55.592610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:55.592248 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:29:56.591835 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:56.591805 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:29:56.592000 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:56.591857 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:29:57.592149 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:57.592116 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:29:57.592641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:57.592168 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:29:58.592716 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:58.592678 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:29:58.593112 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:58.592743 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 22:29:59.013388 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:29:59.013345 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:29:59.596280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:59.596248 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:59.596798 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:59.596778 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:59.597429 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:59.597405 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:29:59.756457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:29:59.756407 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" podStartSLOduration=8.75639401 podStartE2EDuration="8.75639401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:53.10667148 +0000 UTC m=+1312.518857497" watchObservedRunningTime="2026-11-03 22:29:59.75639401 +0000 UTC m=+1319.168580027" Nov 03 22:30:06.756335 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.756300 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq"] Nov 03 22:30:06.761998 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.761975 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.824409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.824377 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq"] Nov 03 22:30:06.849355 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.849314 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.849581 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.849455 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.849581 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.849488 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.849581 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.849535 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.849781 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.849599 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fad8571-4357-4318-8f8a-6515d4211a9a-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.849835 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.849787 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzd6g\" (UniqueName: \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-kube-api-access-pzd6g\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.849896 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.849840 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.849953 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.849936 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.850008 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.849968 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951152 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951122 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzd6g\" (UniqueName: \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-kube-api-access-pzd6g\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951347 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951162 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951347 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951196 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951347 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951215 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951347 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951285 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951347 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951330 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951659 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951359 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951659 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951518 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951659 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951597 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fad8571-4357-4318-8f8a-6515d4211a9a-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951678 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951731 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951934 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951833 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.951990 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.951935 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.952451 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.952428 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fad8571-4357-4318-8f8a-6515d4211a9a-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.953653 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.953636 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:06.954177 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:06.954156 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:07.005018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.004982 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:07.005211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.005186 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzd6g\" (UniqueName: \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-kube-api-access-pzd6g\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:07.075890 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.075828 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:07.360415 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.360316 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x"] Nov 03 22:30:07.366016 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.365994 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.443734 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.443700 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x"] Nov 03 22:30:07.459126 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.459097 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.459292 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.459135 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.459292 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.459222 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.459425 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.459296 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.459425 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.459355 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcf24\" (UniqueName: \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-kube-api-access-tcf24\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.459573 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.459459 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.459573 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.459520 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.459680 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.459579 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d19de6-9f35-42a3-86a2-bfe70247b928-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.459740 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.459707 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.537365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.537324 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq"] Nov 03 22:30:07.543351 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:30:07.543314 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fad8571_4357_4318_8f8a_6515d4211a9a.slice/crio-7f808366b4f18d7cc36365dc4aaa34b55427c4cb7aa1d1384b59503db84e4a7a WatchSource:0}: Error finding container 7f808366b4f18d7cc36365dc4aaa34b55427c4cb7aa1d1384b59503db84e4a7a: Status 404 returned error can't find the container with id 7f808366b4f18d7cc36365dc4aaa34b55427c4cb7aa1d1384b59503db84e4a7a Nov 03 22:30:07.546094 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.546063 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:07.546195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.546132 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:07.546195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.546167 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:07.560932 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.560908 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561007 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.560980 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561007 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561005 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561104 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561043 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561104 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561071 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561104 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561100 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcf24\" (UniqueName: \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-kube-api-access-tcf24\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561175 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561218 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561361 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561259 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d19de6-9f35-42a3-86a2-bfe70247b928-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561460 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561437 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561552 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561489 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.561610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561552 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.562012 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.561986 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.562108 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.562030 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d19de6-9f35-42a3-86a2-bfe70247b928-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.563545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.563516 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.563632 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.563614 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.619099 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.619073 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m"] Nov 03 22:30:07.622401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.622378 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.622676 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.622654 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcf24\" (UniqueName: \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-kube-api-access-tcf24\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.624144 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.624128 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.663251 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.663218 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.663400 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.663378 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.663468 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.663443 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.663898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.663631 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.663898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.663664 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.663898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.663702 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07f28fd-b732-4a25-ac79-2381cea9a252-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.664055 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.663893 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.664055 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.663959 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.664055 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.663991 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tc67\" (UniqueName: \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-kube-api-access-4tc67\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.680235 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.680198 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:07.688602 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.688572 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m"] Nov 03 22:30:07.767548 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.767367 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.767548 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.767441 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.768280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.768229 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.770411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.768418 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.770411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.768463 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07f28fd-b732-4a25-ac79-2381cea9a252-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.770411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.768573 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.770411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.768617 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.770411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.768645 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tc67\" (UniqueName: \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-kube-api-access-4tc67\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.770411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.768744 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.770411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.769772 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.770411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.769999 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.771445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.771420 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.772342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.772312 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07f28fd-b732-4a25-ac79-2381cea9a252-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.772658 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.772635 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.772870 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.772843 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.773020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.772995 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.825600 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.825543 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tc67\" (UniqueName: \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-kube-api-access-4tc67\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.827148 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.827121 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:07.939459 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:07.939427 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:08.000133 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.000085 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" event={"ID":"5fad8571-4357-4318-8f8a-6515d4211a9a","Type":"ContainerStarted","Data":"fe8ead82f2a43bf55a583236a7ac1baf0ccbc38e23ef2eba86f40e93eac8bc88"} Nov 03 22:30:08.000540 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.000161 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" event={"ID":"5fad8571-4357-4318-8f8a-6515d4211a9a","Type":"ContainerStarted","Data":"7f808366b4f18d7cc36365dc4aaa34b55427c4cb7aa1d1384b59503db84e4a7a"} Nov 03 22:30:08.076908 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.076875 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:08.078295 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.078267 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:30:08.078433 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.078315 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:30:08.095456 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.095426 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x"] Nov 03 22:30:08.099364 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:30:08.099337 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d19de6_9f35_42a3_86a2_bfe70247b928.slice/crio-ac6bafc9f836bb2cc6bd7ff3781beab5ec5bafa82cdaad173cae4a325fe44fc9 WatchSource:0}: Error finding container ac6bafc9f836bb2cc6bd7ff3781beab5ec5bafa82cdaad173cae4a325fe44fc9: Status 404 returned error can't find the container with id ac6bafc9f836bb2cc6bd7ff3781beab5ec5bafa82cdaad173cae4a325fe44fc9 Nov 03 22:30:08.102122 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.102092 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:08.102226 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.102154 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:08.102226 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.102195 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:08.115820 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.115765 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podStartSLOduration=2.115747472 podStartE2EDuration="2.115747472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:08.111829093 +0000 UTC m=+1327.524015109" watchObservedRunningTime="2026-11-03 22:30:08.115747472 +0000 UTC m=+1327.527933490" Nov 03 22:30:08.328691 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.328564 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m"] Nov 03 22:30:08.342861 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.342818 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:08.342979 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.342893 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:08.342979 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.342927 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:08.359690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.359659 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h"] Nov 03 22:30:08.365896 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.365872 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.460605 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.454905 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h"] Nov 03 22:30:08.502024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.501457 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.502024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.501549 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.502024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.501600 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.502024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.501678 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47fc9be-c015-4395-91fb-94ecfe8ea648-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.502024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.501700 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.502024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.501740 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.502024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.501768 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.502024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.501791 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6vh5\" (UniqueName: \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-kube-api-access-q6vh5\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.502024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.501830 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.603539 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.603449 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47fc9be-c015-4395-91fb-94ecfe8ea648-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.603539 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.603505 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.603778 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.603568 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.603778 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.603595 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.603778 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.603630 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6vh5\" (UniqueName: \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-kube-api-access-q6vh5\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.603778 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.603690 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.603983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.603819 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.603983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.603886 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.603983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.603958 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.604366 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.604333 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47fc9be-c015-4395-91fb-94ecfe8ea648-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.604453 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.604369 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.604453 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.604439 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.604561 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.604470 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.604665 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.604645 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.606858 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.606840 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.607059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.607043 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.683562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.678777 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6vh5\" (UniqueName: \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-kube-api-access-q6vh5\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.683562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.682771 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:08.697898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:08.697861 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:09.009123 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.009026 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" event={"ID":"a4d19de6-9f35-42a3-86a2-bfe70247b928","Type":"ContainerStarted","Data":"ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd"} Nov 03 22:30:09.009123 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.009088 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" event={"ID":"a4d19de6-9f35-42a3-86a2-bfe70247b928","Type":"ContainerStarted","Data":"ac6bafc9f836bb2cc6bd7ff3781beab5ec5bafa82cdaad173cae4a325fe44fc9"} Nov 03 22:30:09.010995 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.010956 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" event={"ID":"f07f28fd-b732-4a25-ac79-2381cea9a252","Type":"ContainerStarted","Data":"dbf7a0ab5c3a40136ffe38e28eb967616fe177442018703b32f216ec56f4f397"} Nov 03 22:30:09.011139 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.011021 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" event={"ID":"f07f28fd-b732-4a25-ac79-2381cea9a252","Type":"ContainerStarted","Data":"4d70e1610ae8c598b13b87aededcb849170e73eab64f89f0870c29284bef2897"} Nov 03 22:30:09.076631 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.076587 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:30:09.076826 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.076649 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:30:09.114328 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.114288 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h"] Nov 03 22:30:09.169317 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.169264 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podStartSLOduration=2.169231906 podStartE2EDuration="2.169231906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:09.127811263 +0000 UTC m=+1328.539997282" watchObservedRunningTime="2026-11-03 22:30:09.169231906 +0000 UTC m=+1328.581417936" Nov 03 22:30:09.171658 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.171615 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:09.171909 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.171885 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:09.172209 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.172188 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:09.333017 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.313819 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podStartSLOduration=2.313796505 podStartE2EDuration="2.313796505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:09.298808636 +0000 UTC m=+1328.710994667" watchObservedRunningTime="2026-11-03 22:30:09.313796505 +0000 UTC m=+1328.725982523" Nov 03 22:30:09.686478 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.686445 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:09.686912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.686885 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:30:09.687076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:09.687052 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:30:10.036290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.035569 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:10.063205 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.039586 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.063205 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.039644 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:30:10.109519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.108874 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.109519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.108933 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:30:10.164620 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.164581 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" event={"ID":"f47fc9be-c015-4395-91fb-94ecfe8ea648","Type":"ContainerStarted","Data":"a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b"} Nov 03 22:30:10.164858 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.164840 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" event={"ID":"f47fc9be-c015-4395-91fb-94ecfe8ea648","Type":"ContainerStarted","Data":"5767926f398cee95e1089524f2bbec252b58838ec6cf7cb64e41e4db9c297637"} Nov 03 22:30:10.318919 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.318802 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podStartSLOduration=2.3187818079999998 podStartE2EDuration="2.318781808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:10.318031975 +0000 UTC m=+1329.730217985" watchObservedRunningTime="2026-11-03 22:30:10.318781808 +0000 UTC m=+1329.730967826" Nov 03 22:30:10.691520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.686874 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.691520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.687046 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:30:10.775521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.755630 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.775521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.755712 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 22:30:10.775521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.764023 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:10.941519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.940736 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.941519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:10.940798 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:30:11.100580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.100370 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:30:11.100580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.100428 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:30:11.544519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.544179 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:11.544741 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.544534 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 03 22:30:11.560530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.544814 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded" Nov 03 22:30:11.560530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.545357 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:11.622531 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.621397 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 22:30:11.696521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.693979 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:30:11.708522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.704898 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:30:11.708522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.706139 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:11.762527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.760781 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:30:11.762739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.762701 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 22:30:11.787520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.776161 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:11.843571 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.842948 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:11.979314 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.956340 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:30:11.979314 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:11.956401 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:30:12.103946 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.103851 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:30:12.104618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.104551 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:30:12.251521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.230086 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.251521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.236991 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.344523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.329004 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.377630 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.377335 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.448521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.444903 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.552521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.551843 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 22:30:12.552521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.552298 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:30:12.575521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.574666 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.575521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.575100 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.575521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.575441 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:30:12.575521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.575516 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:30:12.632609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.629595 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.632609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.630066 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:30:12.729519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.713186 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.908001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.752071 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:30:12.908001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.760383 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:30:12.908001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.760437 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:30:12.908001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.763058 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.908001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.785605 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:30:12.908001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.785669 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 22:30:12.908001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.833785 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 22:30:12.908001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.834164 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 03 22:30:12.908001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.879098 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:12.908001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:12.879179 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 22:30:13.096952 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.068586 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:30:13.096952 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.068665 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:30:13.102951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.102833 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:30:13.102951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.102892 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:30:13.209516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.208615 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:13.279256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.260707 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:13.330521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.326933 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 22:30:13.361704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.349810 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:13.389538 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.386897 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 22:30:13.465011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.450060 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 22:30:13.697195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.687023 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:13.863308 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.794657 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 22:30:13.863308 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.834422 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:13.863308 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.836570 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 22:30:13.917241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.864284 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 22:30:13.917241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.890872 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 03 22:30:13.917241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.916910 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:30:13.917241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:13.916979 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:30:14.055856 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.055782 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:30:14.056239 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.056176 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:30:14.120629 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.105060 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 03 22:30:14.387284 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.147974 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Nov 03 22:30:14.387284 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.386910 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:30:14.387284 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.386961 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:30:14.469540 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.423760 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:14.469540 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.386819 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 22:30:14.533133 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:30:14.480440 6234 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.717s" Nov 03 22:30:14.706546 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.705551 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:30:14.706546 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.557613 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:14.706546 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.589898 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 22:30:14.706546 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.591053 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 22:30:14.706546 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.591115 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:14.706546 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.705311 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:14.707198 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.706718 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:30:14.707198 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.706763 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 22:30:14.707198 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.707096 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:30:14.707198 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.707133 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 22:30:14.759538 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.759357 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:30:14.759538 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.759432 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:30:14.855523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.855056 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:14.855523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.855222 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 22:30:14.855523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.855330 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:14.882058 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.882014 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 22:30:14.972843 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.972740 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 03 22:30:15.082519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.078884 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.082519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.079039 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:30:15.136520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.111732 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 03 22:30:15.136520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.117568 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:15.136520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:14.957136 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:30:15.136520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.123085 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:30:15.136520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.123135 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:30:15.286480 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.286376 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.321522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.314583 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:30:15.352380 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.352346 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:30:15.375657 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.375621 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:30:15.375893 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.375880 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:30:15.376043 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.376032 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:30:15.535524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.443881 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:15.535524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.456597 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:15.747520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.746847 6234 patch_prober.go:29] interesting pod/metrics-server-779ff6cb48-8rzf9 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.15:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:15.747520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.746903 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-779ff6cb48-8rzf9" podUID="5692356c-6609-46d4-9cd7-513a0d883779" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.15:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:15.777527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.776628 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 22:30:15.777527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.777007 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:15.777527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.777271 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:15.796534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.777908 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 22:30:15.796534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.778418 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 22:30:15.796534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.783807 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.796534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.783851 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 22:30:15.855646 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.854542 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:30:15.905019 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.904967 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 22:30:15.905615 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.905588 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 22:30:15.934544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.934405 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:30:15.958934 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.907069 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.979383 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.979342 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:30:15.979729 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.972218 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:15.985366 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.985341 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.985607 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.985579 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:30:15.985844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:15.985820 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:16.115273 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.115189 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:30:16.227695 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.226448 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.228087 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.227952 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 22:30:16.228300 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.227177 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:16.231509 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.226619 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:30:16.231711 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.231699 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:30:16.234075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.234058 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:30:16.236062 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.236042 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:30:16.236241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.236227 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:30:16.236873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.236855 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:30:16.246314 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.244303 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:30:16.406576 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.392041 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:30:16.578530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.537666 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:30:16.594688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.594657 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:30:16.757451 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.757419 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.757793 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.757746 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 22:30:16.759860 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.759838 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.760043 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.760021 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 22:30:16.987918 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.987880 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.988188 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:16.988158 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 22:30:17.766602 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:17.766570 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:17.808594 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:17.803810 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:17.834921 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:17.826677 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:30:17.834921 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:17.826761 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 22:30:17.869531 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:17.864259 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.132787 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:18.132704 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:30:18.247667 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:18.247623 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.255536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:18.248841 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.922522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:18.802603 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.922522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:18.802680 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.013578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:18.917704 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:19.013578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:18.930712 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.049273 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.049202 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 22:30:19.051129 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.051096 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 22:30:19.092386 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.090888 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:19.092386 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.090948 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.093806 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.093726 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 22:30:19.098131 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.094166 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 22:30:19.098523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.097582 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 22:30:19.167584 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.142207 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 03 22:30:19.167584 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.144342 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nknft" podUID="98ac88a1-8c4e-4fac-b490-d39fe59a433a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.283934 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.283892 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 22:30:19.284386 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.284357 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.296528 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.285093 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:30:19.296860 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.296838 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:30:19.480529 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.451877 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.994787 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.757413 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 22:30:19.994787 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.783648 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:19.994787 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.809678 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:19.994787 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:19.926626 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:30:20.110518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.062234 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.110518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.063819 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:30:20.110518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.092928 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 22:30:20.110518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.093219 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.139829 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.119673 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 22:30:20.139829 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.120874 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:30:20.139829 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.120959 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:30:20.139829 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.121011 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:30:20.221521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.142886 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:20.443617 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.443572 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.445559 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.149026 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:20.445766 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.445741 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.446485 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.145567 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.452553 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.225638 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 22:30:20.456550 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.225663 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.462944 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.225682 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 22:30:20.470023 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.469994 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:30:20.470274 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.470257 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:30:20.470419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.470407 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:30:20.484520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.292784 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.484787 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.292835 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 22:30:20.485125 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.320648 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.485264 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.225595 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 22:30:20.506477 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.506437 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:30:20.508077 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.507678 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:30:20.602860 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.602818 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Nov 03 22:30:20.715903 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.715812 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:20.716158 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.716126 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.788525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.785221 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.813157 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:20.813112 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.073581 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.068798 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.103086 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.103029 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.103837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.103788 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 22:30:21.124912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.124084 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.292630 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.292580 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.321519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.296019 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.372566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.368617 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:30:21.372566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.368685 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:30:21.399519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.399008 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Nov 03 22:30:21.452902 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.452066 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.456862 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.456833 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.462913 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.462588 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:21.462913 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.462638 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.464295 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.463663 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.513613 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.513568 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.514215 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.514173 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.514664 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.514638 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.515653 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.515630 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.524735 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.524711 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:21.524900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.524877 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.785671 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.785626 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.816435 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.816394 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 22:30:21.841532 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.835155 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:30:21.889534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.888971 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:30:21.923512 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:21.896535 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:30:22.071947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.071854 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:22.124462 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.084459 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:30:22.159541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.137166 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:30:22.202568 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.202540 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:30:22.210633 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.210612 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:22.210884 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.210861 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:30:22.228827 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.227887 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" Nov 03 22:30:22.228827 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.227937 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:30:22.228827 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.227971 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:30:22.228827 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.227988 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:22.228827 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.228043 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:30:22.351668 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.340347 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:30:22.358516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.357698 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:30:22.364964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.364938 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:30:22.504031 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.503989 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:22.504219 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:22.504054 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:23.025817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:23.025788 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:23.026730 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:23.026703 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:23.028533 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:23.028509 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:30:29.005890 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:30:29.005845 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:30:52.217059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.217025 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl"] Nov 03 22:30:52.221860 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.221836 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.272270 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.272246 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:30:52.318857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.318827 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:30:52.319047 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.318878 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qcrf7\"" Nov 03 22:30:52.319047 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.318887 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:30:52.322993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.322967 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl"] Nov 03 22:30:52.352822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.352781 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.352822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.352828 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.353070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.352957 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.353139 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.353083 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fm9s\" (UniqueName: \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-kube-api-access-6fm9s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.353139 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.353126 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.353252 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.353231 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.353395 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.353373 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.455304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.455271 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.455527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.455319 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.455527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.455397 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.455527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.455460 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.455527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.455481 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.455767 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.455563 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.455767 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.455685 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fm9s\" (UniqueName: \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-kube-api-access-6fm9s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.456463 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.456434 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.458115 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.458082 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.458612 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.458588 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.458935 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.458915 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.459006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.458930 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.509605 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.509542 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.510101 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.510083 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fm9s\" (UniqueName: \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-kube-api-access-6fm9s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fzzsl\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.537048 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.537011 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:52.756975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.756920 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:30:52.965645 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.965618 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl"] Nov 03 22:30:52.969142 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:30:52.969106 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4b2a00_37a7_4c3b_ae3a_a901ae71815f.slice/crio-473b419ebc74dbeff5cfbed609fe8fd44c665dda5e2cf45265614ae1f32ceaed WatchSource:0}: Error finding container 473b419ebc74dbeff5cfbed609fe8fd44c665dda5e2cf45265614ae1f32ceaed: Status 404 returned error can't find the container with id 473b419ebc74dbeff5cfbed609fe8fd44c665dda5e2cf45265614ae1f32ceaed Nov 03 22:30:52.972805 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.972771 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:52.972915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:52.972851 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:53.924520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:53.920783 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" event={"ID":"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f","Type":"ContainerStarted","Data":"7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe"} Nov 03 22:30:53.924520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:53.920847 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" event={"ID":"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f","Type":"ContainerStarted","Data":"473b419ebc74dbeff5cfbed609fe8fd44c665dda5e2cf45265614ae1f32ceaed"} Nov 03 22:30:53.924520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:53.921918 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:55.032520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:55.031573 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 03 22:30:55.170667 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:55.170613 6234 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fzzsl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:30:55.171018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:55.170991 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" podUID="2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:30:55.260518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:55.249665 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:55.261615 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:55.261572 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8" podUID="48fcdc24-884a-482b-80cb-790ff881816d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Nov 03 22:30:55.818516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:55.818456 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:56.191578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:56.159078 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:56.315522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:56.224964 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:56.499950 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:56.498958 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:30:56.807609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:30:56.741653 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" podStartSLOduration=4.741631582 podStartE2EDuration="4.741631582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:54.040181757 +0000 UTC m=+1373.452367778" watchObservedRunningTime="2026-11-03 22:30:56.741631582 +0000 UTC m=+1376.153817600" Nov 03 22:30:59.010456 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:30:59.010413 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:31:23.248516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.248465 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd"] Nov 03 22:31:23.253185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.253159 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.344004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.343969 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.344191 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.344028 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.344191 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.344133 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xrp\" (UniqueName: \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-kube-api-access-t7xrp\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.344326 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.344185 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.344326 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.344237 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.344326 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.344317 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d34efe6-93ed-4dc1-ba14-896977c661b2-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.344534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.344386 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.344534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.344409 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.344534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.344530 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.443159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.443123 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd"] Nov 03 22:31:23.446159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446125 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xrp\" (UniqueName: \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-kube-api-access-t7xrp\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.446318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446189 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.446318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446243 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.446445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446315 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d34efe6-93ed-4dc1-ba14-896977c661b2-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.446445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446381 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.446445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446417 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.446618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446557 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.446673 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446615 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.446727 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446683 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.446785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446765 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.446904 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.446853 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.447148 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.447124 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.447303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.447275 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d34efe6-93ed-4dc1-ba14-896977c661b2-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.447380 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.447367 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.449292 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.449265 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.451290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.451266 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.500320 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.500243 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xrp\" (UniqueName: \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-kube-api-access-t7xrp\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.500320 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.500260 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.567752 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.567709 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:23.643276 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.643246 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b"] Nov 03 22:31:23.650783 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.650078 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.709338 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.708953 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b"] Nov 03 22:31:23.751348 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.751277 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8219d9c-58a1-4588-a298-7066d5df5d1f-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.751348 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.751319 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.751548 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.751425 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.751548 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.751523 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qst4t\" (UniqueName: \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-kube-api-access-qst4t\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.751622 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.751551 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.751622 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.751582 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.751684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.751627 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.751719 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.751699 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.751754 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.751743 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.852774 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.852733 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8219d9c-58a1-4588-a298-7066d5df5d1f-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.852945 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.852781 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.852945 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.852836 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.852945 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.852918 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qst4t\" (UniqueName: \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-kube-api-access-qst4t\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.853118 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.852955 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.853118 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.852999 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.853118 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.853025 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.853118 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.853089 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.853335 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.853154 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.853335 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.853210 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.853549 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.853490 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.853549 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.853536 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.853692 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.853594 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8219d9c-58a1-4588-a298-7066d5df5d1f-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.853758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.853740 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.855210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.855189 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.855710 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.855693 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.907408 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.907369 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.907755 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.907734 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qst4t\" (UniqueName: \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-kube-api-access-qst4t\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.966076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.966044 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:23.996176 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:23.996140 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd"] Nov 03 22:31:24.000132 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:31:24.000107 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d34efe6_93ed_4dc1_ba14_896977c661b2.slice/crio-83276565c310b9063b3a4e8349577deb9c1aadfcf955710d71d841199dc1a3b8 WatchSource:0}: Error finding container 83276565c310b9063b3a4e8349577deb9c1aadfcf955710d71d841199dc1a3b8: Status 404 returned error can't find the container with id 83276565c310b9063b3a4e8349577deb9c1aadfcf955710d71d841199dc1a3b8 Nov 03 22:31:24.003830 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.003790 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:31:24.003923 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.003881 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:31:24.003990 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.003947 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:31:24.427251 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.427166 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b"] Nov 03 22:31:24.431909 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:31:24.431877 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8219d9c_58a1_4588_a298_7066d5df5d1f.slice/crio-5f096d939981e213019ae71a54703b3f96ad1b8762d853188dbdb0c61030275e WatchSource:0}: Error finding container 5f096d939981e213019ae71a54703b3f96ad1b8762d853188dbdb0c61030275e: Status 404 returned error can't find the container with id 5f096d939981e213019ae71a54703b3f96ad1b8762d853188dbdb0c61030275e Nov 03 22:31:24.434835 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.434801 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:31:24.434946 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.434884 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:31:24.434946 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.434929 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:31:24.530388 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.530307 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" event={"ID":"3d34efe6-93ed-4dc1-ba14-896977c661b2","Type":"ContainerStarted","Data":"70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d"} Nov 03 22:31:24.530558 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.530425 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" event={"ID":"3d34efe6-93ed-4dc1-ba14-896977c661b2","Type":"ContainerStarted","Data":"83276565c310b9063b3a4e8349577deb9c1aadfcf955710d71d841199dc1a3b8"} Nov 03 22:31:24.535211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.535180 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" event={"ID":"a8219d9c-58a1-4588-a298-7066d5df5d1f","Type":"ContainerStarted","Data":"5f096d939981e213019ae71a54703b3f96ad1b8762d853188dbdb0c61030275e"} Nov 03 22:31:24.568680 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:24.568648 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:25.574517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:25.573595 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:31:25.574517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:25.573655 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 22:31:25.616562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:25.616487 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" event={"ID":"a8219d9c-58a1-4588-a298-7066d5df5d1f","Type":"ContainerStarted","Data":"e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce"} Nov 03 22:31:25.741525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:25.740607 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" podStartSLOduration=2.7405856330000002 podStartE2EDuration="2.740585633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:24.650325331 +0000 UTC m=+1404.062511349" watchObservedRunningTime="2026-11-03 22:31:25.740585633 +0000 UTC m=+1405.152771662" Nov 03 22:31:25.749562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:25.742794 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" podStartSLOduration=2.742781548 podStartE2EDuration="2.742781548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:25.740120079 +0000 UTC m=+1405.152306097" watchObservedRunningTime="2026-11-03 22:31:25.742781548 +0000 UTC m=+1405.154967588" Nov 03 22:31:25.968144 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:25.968114 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:25.968475 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:25.968455 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:31:25.968651 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:25.968624 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 22:31:26.589978 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:26.589934 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:31:26.600980 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:26.600941 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 22:31:26.973519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:26.972541 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:31:26.973519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:26.972606 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 22:31:27.624521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:27.611158 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:31:27.625623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:27.625577 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 22:31:27.974520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:27.973617 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:31:27.974520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:27.973695 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 22:31:28.569519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:28.568622 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:31:28.569519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:28.568688 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 22:31:28.992520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:28.984227 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:31:28.992520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:28.984284 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 22:31:29.029616 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:31:29.029579 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:31:29.590908 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:29.586702 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:31:29.590908 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:29.586784 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 22:31:29.990518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:29.982008 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:31:29.990518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:29.982068 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 22:31:30.568232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:30.568195 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:31:30.568864 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:30.568825 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 22:31:30.970973 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:30.970932 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:31:30.971272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:30.971239 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 22:31:31.913222 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:31.913190 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:31.914805 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:31.914780 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:31.915476 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:31.915444 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:31:31.967374 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:31.967339 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:31:31.967593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:31.967393 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 22:31:33.829418 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:33.829375 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:33.829983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:33.829962 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:33.831155 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:31:33.831130 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:31:58.997796 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:31:58.997756 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:32:08.058523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:32:08.055612 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:32:28.997581 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:32:28.997476 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:32:58.998338 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:32:58.998298 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:33:29.002044 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:33:29.002008 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:33:44.436597 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:33:44.436562 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:33:58.997552 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:33:58.997513 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:34:28.997311 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:34:28.997272 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:34:58.998458 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:34:58.998414 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:35:08.813584 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:08.813547 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx"] Nov 03 22:35:08.814209 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:08.813817 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" containerName="istio-proxy" containerID="cri-o://8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e" gracePeriod=30 Nov 03 22:35:14.089659 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.089604 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:35:14.160792 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.160759 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-envoy\") pod \"252da87c-dca5-447a-b98f-ad65e61b8337\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " Nov 03 22:35:14.160963 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.160808 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252da87c-dca5-447a-b98f-ad65e61b8337-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252da87c-dca5-447a-b98f-ad65e61b8337-istiod-ca-cert\") pod \"252da87c-dca5-447a-b98f-ad65e61b8337\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " Nov 03 22:35:14.160963 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.160882 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-certs\") pod \"252da87c-dca5-447a-b98f-ad65e61b8337\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " Nov 03 22:35:14.161082 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161019 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/252da87c-dca5-447a-b98f-ad65e61b8337-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252da87c-dca5-447a-b98f-ad65e61b8337-istio-podinfo\") pod \"252da87c-dca5-447a-b98f-ad65e61b8337\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " Nov 03 22:35:14.161136 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161092 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-credential-socket\") pod \"252da87c-dca5-447a-b98f-ad65e61b8337\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " Nov 03 22:35:14.161216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161181 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-certs" pod "252da87c-dca5-447a-b98f-ad65e61b8337" (UID: "252da87c-dca5-447a-b98f-ad65e61b8337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.161216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161188 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-data\") pod \"252da87c-dca5-447a-b98f-ad65e61b8337\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " Nov 03 22:35:14.161333 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161266 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252da87c-dca5-447a-b98f-ad65e61b8337-istiod-ca-cert" pod "252da87c-dca5-447a-b98f-ad65e61b8337" (UID: "252da87c-dca5-447a-b98f-ad65e61b8337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:14.161333 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161295 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-kube-api-access-wsq2k\" (UniqueName: \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-kube-api-access-wsq2k\") pod \"252da87c-dca5-447a-b98f-ad65e61b8337\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " Nov 03 22:35:14.161439 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161399 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-socket\") pod \"252da87c-dca5-447a-b98f-ad65e61b8337\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " Nov 03 22:35:14.161525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161399 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-data" pod "252da87c-dca5-447a-b98f-ad65e61b8337" (UID: "252da87c-dca5-447a-b98f-ad65e61b8337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.161592 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161414 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-credential-socket" pod "252da87c-dca5-447a-b98f-ad65e61b8337" (UID: "252da87c-dca5-447a-b98f-ad65e61b8337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.161592 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161478 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-istio-token\" (UniqueName: \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-istio-token\") pod \"252da87c-dca5-447a-b98f-ad65e61b8337\" (UID: \"252da87c-dca5-447a-b98f-ad65e61b8337\") " Nov 03 22:35:14.161696 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.161643 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-socket" pod "252da87c-dca5-447a-b98f-ad65e61b8337" (UID: "252da87c-dca5-447a-b98f-ad65e61b8337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.162257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.162240 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.162295 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.162259 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.162295 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.162269 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.162295 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.162281 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.162412 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.162306 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252da87c-dca5-447a-b98f-ad65e61b8337-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.164640 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.164609 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/252da87c-dca5-447a-b98f-ad65e61b8337-istio-podinfo" pod "252da87c-dca5-447a-b98f-ad65e61b8337" (UID: "252da87c-dca5-447a-b98f-ad65e61b8337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:14.164640 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.164632 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-kube-api-access-wsq2k" pod "252da87c-dca5-447a-b98f-ad65e61b8337" (UID: "252da87c-dca5-447a-b98f-ad65e61b8337"). InnerVolumeSpecName "kube-api-access-wsq2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:14.164786 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.164659 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-istio-token" pod "252da87c-dca5-447a-b98f-ad65e61b8337" (UID: "252da87c-dca5-447a-b98f-ad65e61b8337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:14.164786 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.164670 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-envoy" pod "252da87c-dca5-447a-b98f-ad65e61b8337" (UID: "252da87c-dca5-447a-b98f-ad65e61b8337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.263887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.263856 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252da87c-dca5-447a-b98f-ad65e61b8337-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.263887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.263886 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252da87c-dca5-447a-b98f-ad65e61b8337-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.264079 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.263900 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsq2k\" (UniqueName: \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-kube-api-access-wsq2k\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.264079 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.263914 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252da87c-dca5-447a-b98f-ad65e61b8337-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.461250 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.461208 6234 generic.go:356] "Generic (PLEG): container finished" podID="252da87c-dca5-447a-b98f-ad65e61b8337" containerID="8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e" exitCode=0 Nov 03 22:35:14.461764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.461476 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" event={"ID":"252da87c-dca5-447a-b98f-ad65e61b8337","Type":"ContainerDied","Data":"8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e"} Nov 03 22:35:14.461946 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.461927 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" event={"ID":"252da87c-dca5-447a-b98f-ad65e61b8337","Type":"ContainerDied","Data":"ddf0fa526726dabbbbd004e020b5e1fce147617ecc09bb1e3222e347d38e0dcd"} Nov 03 22:35:14.462040 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.461585 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx" Nov 03 22:35:14.462040 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.461968 6234 scope.go:122] "RemoveContainer" containerID="8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e" Nov 03 22:35:14.475759 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.475732 6234 scope.go:122] "RemoveContainer" containerID="8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e" Nov 03 22:35:14.476114 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:35:14.476091 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e\": container with ID starting with 8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e not found: ID does not exist" containerID="8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e" Nov 03 22:35:14.476198 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.476125 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e"} err="failed to get container status \"8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e\": rpc error: code = NotFound desc = could not find container \"8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e\": container with ID starting with 8b1984d9e2dead38dd824c3907eb0e3f3d6dc5350cd729a467a4bcc52437f64e not found: ID does not exist" Nov 03 22:35:14.617445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.617401 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx"] Nov 03 22:35:14.620831 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.620800 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-5krcx"] Nov 03 22:35:14.707057 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.707025 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" path="/var/lib/kubelet/pods/252da87c-dca5-447a-b98f-ad65e61b8337/volumes" Nov 03 22:35:14.750531 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.750440 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772"] Nov 03 22:35:14.751266 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.751242 6234 cpu_manager.go:414] "containerMap: removing container" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" containerName="istio-proxy" Nov 03 22:35:14.751266 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.751264 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" containerName="istio-proxy" Nov 03 22:35:14.751509 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.751480 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="252da87c-dca5-447a-b98f-ad65e61b8337" containerName="istio-proxy" Nov 03 22:35:14.757004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.756984 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.810455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.810427 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772"] Nov 03 22:35:14.873923 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.873883 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f58c897-de13-4ef8-ae65-7d343ec39dff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.874128 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.873951 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.874211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.874177 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.874276 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.874226 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.874342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.874281 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.874403 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.874365 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqfw\" (UniqueName: \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-kube-api-access-bgqfw\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.874403 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.874401 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.874546 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.874529 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.874633 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.874554 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.975799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.975763 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.975799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.975803 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.975832 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.975878 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqfw\" (UniqueName: \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-kube-api-access-bgqfw\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.975919 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.976035 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976250 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.976061 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976250 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.976206 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976250 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.976213 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f58c897-de13-4ef8-ae65-7d343ec39dff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.976267 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976460 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.976404 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976659 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.976632 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.976807 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.976789 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.977262 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.977238 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f58c897-de13-4ef8-ae65-7d343ec39dff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.978637 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.978610 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:14.978793 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:14.978777 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:15.029466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:15.029398 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:15.029624 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:15.029489 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqfw\" (UniqueName: \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-kube-api-access-bgqfw\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zc772\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:15.071651 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:15.071626 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:15.496887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:15.496856 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772"] Nov 03 22:35:15.501105 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:35:15.501079 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f58c897_de13_4ef8_ae65_7d343ec39dff.slice/crio-c2da560ad44a5c754a7e362c9b23e6bfcecacb66e20836dd2cfb3ac85fd38c6d WatchSource:0}: Error finding container c2da560ad44a5c754a7e362c9b23e6bfcecacb66e20836dd2cfb3ac85fd38c6d: Status 404 returned error can't find the container with id c2da560ad44a5c754a7e362c9b23e6bfcecacb66e20836dd2cfb3ac85fd38c6d Nov 03 22:35:15.503730 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:15.503697 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:35:15.503824 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:15.503781 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:35:15.503824 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:15.503819 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:35:16.381063 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:16.379793 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:35:16.532516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:16.528811 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" event={"ID":"4f58c897-de13-4ef8-ae65-7d343ec39dff","Type":"ContainerStarted","Data":"c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf"} Nov 03 22:35:16.532516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:16.528929 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" event={"ID":"4f58c897-de13-4ef8-ae65-7d343ec39dff","Type":"ContainerStarted","Data":"c2da560ad44a5c754a7e362c9b23e6bfcecacb66e20836dd2cfb3ac85fd38c6d"} Nov 03 22:35:17.082516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:17.080779 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:17.084190 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:17.083935 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zc772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:35:17.084190 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:17.084026 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:35:18.072734 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:18.072696 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zc772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:35:18.073169 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:18.072773 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:35:19.071968 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:19.071935 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zc772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:35:19.072158 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:19.071995 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:35:20.083678 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:20.083640 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zc772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:35:20.084250 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:20.084214 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:35:21.071912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:21.071883 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zc772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:35:21.072195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:21.072167 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:35:22.072168 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:22.072130 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zc772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:35:22.072619 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:22.072205 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:35:23.072747 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:23.072697 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zc772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:35:23.073131 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:23.072787 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 22:35:24.408367 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:24.408321 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:24.408891 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:24.408705 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:24.410039 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:24.410005 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:35:24.566679 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:35:24.566628 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" podStartSLOduration=10.566614752 podStartE2EDuration="10.566614752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:16.697854948 +0000 UTC m=+1636.110040966" watchObservedRunningTime="2026-11-03 22:35:24.566614752 +0000 UTC m=+1643.978800769" Nov 03 22:35:28.997225 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:35:28.997192 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:35:59.005411 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:35:59.005375 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:36:29.014234 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:36:29.014057 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:36:30.376926 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.376894 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp"] Nov 03 22:36:30.381336 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.381318 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.483455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.483424 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp"] Nov 03 22:36:30.486361 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.486327 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.486550 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.486480 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq4cj\" (UniqueName: \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-kube-api-access-bq4cj\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.486640 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.486548 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.486702 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.486648 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.486702 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.486675 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.486824 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.486748 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.486824 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.486769 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.486929 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.486867 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.487078 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.487047 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.588373 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.588340 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.588609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.588442 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.588609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.588540 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq4cj\" (UniqueName: \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-kube-api-access-bq4cj\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.588609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.588572 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.588609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.588592 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.588848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.588629 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.588848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.588700 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.588848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.588727 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.588848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.588781 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.589070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.588978 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.589125 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.589066 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.589125 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.589114 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.589239 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.589192 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.589537 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.589478 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.590655 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.590634 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.591058 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.591040 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.643234 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.643152 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.643400 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.643377 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq4cj\" (UniqueName: \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-kube-api-access-bq4cj\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4vztp\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:30.695085 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:30.695051 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:31.124761 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:31.124723 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp"] Nov 03 22:36:31.129746 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:36:31.129716 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017c1e6a_eb77_43f5_bdaf_e80ed469ddcd.slice/crio-00cd983792cd7c4212f3155219afd904b6a1a1bddd3fea639316f6b0af97838a WatchSource:0}: Error finding container 00cd983792cd7c4212f3155219afd904b6a1a1bddd3fea639316f6b0af97838a: Status 404 returned error can't find the container with id 00cd983792cd7c4212f3155219afd904b6a1a1bddd3fea639316f6b0af97838a Nov 03 22:36:31.132572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:31.132539 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:36:31.132663 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:31.132610 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:36:31.132663 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:31.132640 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:36:31.176648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:31.176619 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" event={"ID":"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd","Type":"ContainerStarted","Data":"00cd983792cd7c4212f3155219afd904b6a1a1bddd3fea639316f6b0af97838a"} Nov 03 22:36:32.185192 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:32.185157 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" event={"ID":"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd","Type":"ContainerStarted","Data":"1c89b1a59bfd32c6c7f6d3f078a8282fa675c41a24d1197104e6a842cf12ab84"} Nov 03 22:36:32.352557 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:32.351552 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podStartSLOduration=2.351533105 podStartE2EDuration="2.351533105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:32.350435558 +0000 UTC m=+1711.762621577" watchObservedRunningTime="2026-11-03 22:36:32.351533105 +0000 UTC m=+1711.763719121" Nov 03 22:36:32.697468 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:32.697435 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:32.704269 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:32.704228 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4vztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:36:32.704457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:32.704296 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:36:33.696155 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:33.696121 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4vztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:36:33.696618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:33.696174 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:36:34.695906 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:34.695874 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4vztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:36:34.696179 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:34.695923 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:36:35.696131 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:35.696091 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4vztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:36:35.696310 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:35.696153 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:36:36.696133 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:36.696097 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4vztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:36:36.696307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:36.696156 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:36:37.695658 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:37.695619 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4vztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:36:37.696135 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:37.695679 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:36:38.695895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:38.695860 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4vztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:36:38.696323 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:38.695913 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 22:36:39.895598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:39.894847 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq"] Nov 03 22:36:39.895598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:39.895116 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" containerID="cri-o://fe8ead82f2a43bf55a583236a7ac1baf0ccbc38e23ef2eba86f40e93eac8bc88" gracePeriod=30 Nov 03 22:36:39.935854 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:39.935819 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:39.939447 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:39.939418 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:39.940324 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:39.940304 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:36:40.715834 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:40.715796 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d"] Nov 03 22:36:40.716070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:40.716043 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" containerID="cri-o://bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8" gracePeriod=30 Nov 03 22:36:40.760527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:40.760479 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd"] Nov 03 22:36:40.760770 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:40.760728 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerName="istio-proxy" containerID="cri-o://70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d" gracePeriod=30 Nov 03 22:36:45.289580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.289227 6234 generic.go:356] "Generic (PLEG): container finished" podID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerID="fe8ead82f2a43bf55a583236a7ac1baf0ccbc38e23ef2eba86f40e93eac8bc88" exitCode=0 Nov 03 22:36:45.289580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.289483 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" event={"ID":"5fad8571-4357-4318-8f8a-6515d4211a9a","Type":"ContainerDied","Data":"fe8ead82f2a43bf55a583236a7ac1baf0ccbc38e23ef2eba86f40e93eac8bc88"} Nov 03 22:36:45.372399 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.372369 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:36:45.510837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.510750 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fad8571-4357-4318-8f8a-6515d4211a9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fad8571-4357-4318-8f8a-6515d4211a9a-istiod-ca-cert\") pod \"5fad8571-4357-4318-8f8a-6515d4211a9a\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " Nov 03 22:36:45.511541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.511516 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-certs\") pod \"5fad8571-4357-4318-8f8a-6515d4211a9a\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " Nov 03 22:36:45.511983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.511960 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-podinfo\") pod \"5fad8571-4357-4318-8f8a-6515d4211a9a\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " Nov 03 22:36:45.512369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.512347 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-kube-api-access-pzd6g\" (UniqueName: \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-kube-api-access-pzd6g\") pod \"5fad8571-4357-4318-8f8a-6515d4211a9a\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " Nov 03 22:36:45.512617 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.512567 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-data\") pod \"5fad8571-4357-4318-8f8a-6515d4211a9a\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " Nov 03 22:36:45.512991 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.512974 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-token\" (UniqueName: \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-token\") pod \"5fad8571-4357-4318-8f8a-6515d4211a9a\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " Nov 03 22:36:45.513569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.511408 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fad8571-4357-4318-8f8a-6515d4211a9a-istiod-ca-cert" pod "5fad8571-4357-4318-8f8a-6515d4211a9a" (UID: "5fad8571-4357-4318-8f8a-6515d4211a9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:45.513569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.512047 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-certs" pod "5fad8571-4357-4318-8f8a-6515d4211a9a" (UID: "5fad8571-4357-4318-8f8a-6515d4211a9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:45.513569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.512883 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-data" pod "5fad8571-4357-4318-8f8a-6515d4211a9a" (UID: "5fad8571-4357-4318-8f8a-6515d4211a9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:45.513910 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.513893 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-credential-socket\") pod \"5fad8571-4357-4318-8f8a-6515d4211a9a\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " Nov 03 22:36:45.514756 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.514342 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-socket\") pod \"5fad8571-4357-4318-8f8a-6515d4211a9a\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " Nov 03 22:36:45.514961 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.514240 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-credential-socket" pod "5fad8571-4357-4318-8f8a-6515d4211a9a" (UID: "5fad8571-4357-4318-8f8a-6515d4211a9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:45.514961 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.514729 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-socket" pod "5fad8571-4357-4318-8f8a-6515d4211a9a" (UID: "5fad8571-4357-4318-8f8a-6515d4211a9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:45.515189 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.515172 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-envoy\") pod \"5fad8571-4357-4318-8f8a-6515d4211a9a\" (UID: \"5fad8571-4357-4318-8f8a-6515d4211a9a\") " Nov 03 22:36:45.517244 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.517222 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.517419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.517402 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.517564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.517548 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.517685 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.517673 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fad8571-4357-4318-8f8a-6515d4211a9a-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.517799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.517788 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.517872 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.517557 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-envoy" pod "5fad8571-4357-4318-8f8a-6515d4211a9a" (UID: "5fad8571-4357-4318-8f8a-6515d4211a9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:45.520756 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.520726 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-kube-api-access-pzd6g" pod "5fad8571-4357-4318-8f8a-6515d4211a9a" (UID: "5fad8571-4357-4318-8f8a-6515d4211a9a"). InnerVolumeSpecName "kube-api-access-pzd6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:45.520857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.520822 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-podinfo" pod "5fad8571-4357-4318-8f8a-6515d4211a9a" (UID: "5fad8571-4357-4318-8f8a-6515d4211a9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:45.522738 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.522709 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-token" pod "5fad8571-4357-4318-8f8a-6515d4211a9a" (UID: "5fad8571-4357-4318-8f8a-6515d4211a9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:45.619300 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.619269 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.619300 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.619299 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzd6g\" (UniqueName: \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-kube-api-access-pzd6g\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.619571 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.619309 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.619571 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:45.619318 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fad8571-4357-4318-8f8a-6515d4211a9a-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.101079 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.101049 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:36:46.171206 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.171180 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:36:46.230683 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.230656 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-data\") pod \"6861036d-82cd-413b-88c2-df43b2b7a613\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " Nov 03 22:36:46.230863 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.230720 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6861036d-82cd-413b-88c2-df43b2b7a613-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6861036d-82cd-413b-88c2-df43b2b7a613-istio-podinfo\") pod \"6861036d-82cd-413b-88c2-df43b2b7a613\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " Nov 03 22:36:46.230863 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.230754 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-kube-api-access-7sz9j\" (UniqueName: \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-kube-api-access-7sz9j\") pod \"6861036d-82cd-413b-88c2-df43b2b7a613\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " Nov 03 22:36:46.230863 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.230802 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-certs\") pod \"6861036d-82cd-413b-88c2-df43b2b7a613\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " Nov 03 22:36:46.231611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.230867 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6861036d-82cd-413b-88c2-df43b2b7a613-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6861036d-82cd-413b-88c2-df43b2b7a613-istiod-ca-cert\") pod \"6861036d-82cd-413b-88c2-df43b2b7a613\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " Nov 03 22:36:46.231611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.230916 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-credential-socket\") pod \"6861036d-82cd-413b-88c2-df43b2b7a613\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " Nov 03 22:36:46.231611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.230971 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-istio-token\" (UniqueName: \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-istio-token\") pod \"6861036d-82cd-413b-88c2-df43b2b7a613\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " Nov 03 22:36:46.231611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.230969 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-data" pod "6861036d-82cd-413b-88c2-df43b2b7a613" (UID: "6861036d-82cd-413b-88c2-df43b2b7a613"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.231611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.231032 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-socket\") pod \"6861036d-82cd-413b-88c2-df43b2b7a613\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " Nov 03 22:36:46.231611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.231216 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-envoy\") pod \"6861036d-82cd-413b-88c2-df43b2b7a613\" (UID: \"6861036d-82cd-413b-88c2-df43b2b7a613\") " Nov 03 22:36:46.231611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.231436 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-socket" pod "6861036d-82cd-413b-88c2-df43b2b7a613" (UID: "6861036d-82cd-413b-88c2-df43b2b7a613"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.231981 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.231623 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-credential-socket" pod "6861036d-82cd-413b-88c2-df43b2b7a613" (UID: "6861036d-82cd-413b-88c2-df43b2b7a613"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.231981 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.231661 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6861036d-82cd-413b-88c2-df43b2b7a613-istiod-ca-cert" pod "6861036d-82cd-413b-88c2-df43b2b7a613" (UID: "6861036d-82cd-413b-88c2-df43b2b7a613"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:46.231981 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.231840 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-certs" pod "6861036d-82cd-413b-88c2-df43b2b7a613" (UID: "6861036d-82cd-413b-88c2-df43b2b7a613"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.232205 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.232181 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.232293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.232209 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.232293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.232226 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6861036d-82cd-413b-88c2-df43b2b7a613-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.232293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.232241 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.232293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.232255 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.234207 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.234030 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-envoy" pod "6861036d-82cd-413b-88c2-df43b2b7a613" (UID: "6861036d-82cd-413b-88c2-df43b2b7a613"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.234207 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.234098 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-istio-token" pod "6861036d-82cd-413b-88c2-df43b2b7a613" (UID: "6861036d-82cd-413b-88c2-df43b2b7a613"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:46.234207 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.234133 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-kube-api-access-7sz9j" pod "6861036d-82cd-413b-88c2-df43b2b7a613" (UID: "6861036d-82cd-413b-88c2-df43b2b7a613"). InnerVolumeSpecName "kube-api-access-7sz9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:46.234207 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.234187 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6861036d-82cd-413b-88c2-df43b2b7a613-istio-podinfo" pod "6861036d-82cd-413b-88c2-df43b2b7a613" (UID: "6861036d-82cd-413b-88c2-df43b2b7a613"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:46.298366 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.298337 6234 generic.go:356] "Generic (PLEG): container finished" podID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerID="70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d" exitCode=0 Nov 03 22:36:46.298794 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.298396 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" Nov 03 22:36:46.298794 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.298415 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" event={"ID":"3d34efe6-93ed-4dc1-ba14-896977c661b2","Type":"ContainerDied","Data":"70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d"} Nov 03 22:36:46.298794 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.298468 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd" event={"ID":"3d34efe6-93ed-4dc1-ba14-896977c661b2","Type":"ContainerDied","Data":"83276565c310b9063b3a4e8349577deb9c1aadfcf955710d71d841199dc1a3b8"} Nov 03 22:36:46.298794 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.298503 6234 scope.go:122] "RemoveContainer" containerID="70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d" Nov 03 22:36:46.299984 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.299964 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" Nov 03 22:36:46.300090 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.299978 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq" event={"ID":"5fad8571-4357-4318-8f8a-6515d4211a9a","Type":"ContainerDied","Data":"7f808366b4f18d7cc36365dc4aaa34b55427c4cb7aa1d1384b59503db84e4a7a"} Nov 03 22:36:46.301313 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.301282 6234 generic.go:356] "Generic (PLEG): container finished" podID="6861036d-82cd-413b-88c2-df43b2b7a613" containerID="bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8" exitCode=0 Nov 03 22:36:46.301435 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.301333 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" Nov 03 22:36:46.301435 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.301368 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" event={"ID":"6861036d-82cd-413b-88c2-df43b2b7a613","Type":"ContainerDied","Data":"bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8"} Nov 03 22:36:46.301435 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.301402 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d" event={"ID":"6861036d-82cd-413b-88c2-df43b2b7a613","Type":"ContainerDied","Data":"df21c10b39562fe6c9c92aed7ca1afef2f0ccade38d4d10b38ae59fd0828079f"} Nov 03 22:36:46.310210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.310187 6234 scope.go:122] "RemoveContainer" containerID="70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d" Nov 03 22:36:46.310552 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:36:46.310454 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d\": container with ID starting with 70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d not found: ID does not exist" containerID="70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d" Nov 03 22:36:46.310552 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.310479 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d"} err="failed to get container status \"70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d\": rpc error: code = NotFound desc = could not find container \"70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d\": container with ID starting with 70f6860668a5f55401a6f3899046cf0b422bfd86c269653c5bd7788aa60e684d not found: ID does not exist" Nov 03 22:36:46.310552 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.310529 6234 scope.go:122] "RemoveContainer" containerID="fe8ead82f2a43bf55a583236a7ac1baf0ccbc38e23ef2eba86f40e93eac8bc88" Nov 03 22:36:46.332906 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.330978 6234 scope.go:122] "RemoveContainer" containerID="bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8" Nov 03 22:36:46.333193 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.333169 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-certs\") pod \"3d34efe6-93ed-4dc1-ba14-896977c661b2\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " Nov 03 22:36:46.333257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.333217 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d34efe6-93ed-4dc1-ba14-896977c661b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d34efe6-93ed-4dc1-ba14-896977c661b2-istiod-ca-cert\") pod \"3d34efe6-93ed-4dc1-ba14-896977c661b2\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " Nov 03 22:36:46.333310 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.333293 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-socket\") pod \"3d34efe6-93ed-4dc1-ba14-896977c661b2\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " Nov 03 22:36:46.333363 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.333350 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-podinfo\") pod \"3d34efe6-93ed-4dc1-ba14-896977c661b2\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " Nov 03 22:36:46.334368 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.333953 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-envoy\") pod \"3d34efe6-93ed-4dc1-ba14-896977c661b2\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " Nov 03 22:36:46.334368 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.334080 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-data\") pod \"3d34efe6-93ed-4dc1-ba14-896977c661b2\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " Nov 03 22:36:46.334368 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.334113 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-credential-socket\") pod \"3d34efe6-93ed-4dc1-ba14-896977c661b2\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " Nov 03 22:36:46.334368 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.334126 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d34efe6-93ed-4dc1-ba14-896977c661b2-istiod-ca-cert" pod "3d34efe6-93ed-4dc1-ba14-896977c661b2" (UID: "3d34efe6-93ed-4dc1-ba14-896977c661b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:46.334368 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.334195 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-kube-api-access-t7xrp\" (UniqueName: \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-kube-api-access-t7xrp\") pod \"3d34efe6-93ed-4dc1-ba14-896977c661b2\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " Nov 03 22:36:46.334368 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.334320 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-token\" (UniqueName: \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-token\") pod \"3d34efe6-93ed-4dc1-ba14-896977c661b2\" (UID: \"3d34efe6-93ed-4dc1-ba14-896977c661b2\") " Nov 03 22:36:46.335114 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.334429 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-certs" pod "3d34efe6-93ed-4dc1-ba14-896977c661b2" (UID: "3d34efe6-93ed-4dc1-ba14-896977c661b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.336203 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.335754 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-credential-socket" pod "3d34efe6-93ed-4dc1-ba14-896977c661b2" (UID: "3d34efe6-93ed-4dc1-ba14-896977c661b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.336779 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.336437 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-data" pod "3d34efe6-93ed-4dc1-ba14-896977c661b2" (UID: "3d34efe6-93ed-4dc1-ba14-896977c661b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.336779 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.336469 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-socket" pod "3d34efe6-93ed-4dc1-ba14-896977c661b2" (UID: "3d34efe6-93ed-4dc1-ba14-896977c661b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.337303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.337161 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6861036d-82cd-413b-88c2-df43b2b7a613-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.337303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.337189 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.337303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.337203 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d34efe6-93ed-4dc1-ba14-896977c661b2-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.337303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.337203 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-envoy" pod "3d34efe6-93ed-4dc1-ba14-896977c661b2" (UID: "3d34efe6-93ed-4dc1-ba14-896977c661b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:46.337303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.337217 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.337303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.337231 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6861036d-82cd-413b-88c2-df43b2b7a613-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.337303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.337245 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sz9j\" (UniqueName: \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-kube-api-access-7sz9j\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.337303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.337259 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6861036d-82cd-413b-88c2-df43b2b7a613-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.337303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.337271 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.337303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.337283 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.338843 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.338818 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-podinfo" pod "3d34efe6-93ed-4dc1-ba14-896977c661b2" (UID: "3d34efe6-93ed-4dc1-ba14-896977c661b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:46.340116 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.340090 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-token" pod "3d34efe6-93ed-4dc1-ba14-896977c661b2" (UID: "3d34efe6-93ed-4dc1-ba14-896977c661b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:46.340654 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.340629 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-kube-api-access-t7xrp" pod "3d34efe6-93ed-4dc1-ba14-896977c661b2" (UID: "3d34efe6-93ed-4dc1-ba14-896977c661b2"). InnerVolumeSpecName "kube-api-access-t7xrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:46.346356 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.346300 6234 scope.go:122] "RemoveContainer" containerID="bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8" Nov 03 22:36:46.346673 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:36:46.346655 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8\": container with ID starting with bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8 not found: ID does not exist" containerID="bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8" Nov 03 22:36:46.346737 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.346682 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8"} err="failed to get container status \"bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8\": rpc error: code = NotFound desc = could not find container \"bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8\": container with ID starting with bb4eea15876598c748ffd3cecc0c273f98ebf50461d0f46095af155eefd283b8 not found: ID does not exist" Nov 03 22:36:46.445107 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.445072 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xrp\" (UniqueName: \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-kube-api-access-t7xrp\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.445107 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.445109 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.445354 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.445124 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.445354 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.445136 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d34efe6-93ed-4dc1-ba14-896977c661b2-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.445354 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.445163 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d"] Nov 03 22:36:46.448800 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.448771 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-66j4d"] Nov 03 22:36:46.550072 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.550041 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq"] Nov 03 22:36:46.553414 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.553386 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-b7gkq"] Nov 03 22:36:46.710326 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.710280 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" path="/var/lib/kubelet/pods/5fad8571-4357-4318-8f8a-6515d4211a9a/volumes" Nov 03 22:36:46.711256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.711228 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" path="/var/lib/kubelet/pods/6861036d-82cd-413b-88c2-df43b2b7a613/volumes" Nov 03 22:36:46.719009 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.718956 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd"] Nov 03 22:36:46.725278 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:46.725247 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-qbfrd"] Nov 03 22:36:48.706415 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:48.706382 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" path="/var/lib/kubelet/pods/3d34efe6-93ed-4dc1-ba14-896977c661b2/volumes" Nov 03 22:36:56.283984 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:56.283940 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz"] Nov 03 22:36:56.284539 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:56.284201 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" containerID="cri-o://dab454577d84c14de1f49bcb15ce8a8df95d917d48aabae1f823df4a718b83ad" gracePeriod=30 Nov 03 22:36:58.691708 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:36:58.691674 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:36:59.002798 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:36:59.002669 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:37:00.448530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:00.445552 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:37:00.448530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:00.445633 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:37:01.423874 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:01.423840 6234 generic.go:356] "Generic (PLEG): container finished" podID="80f18706-2830-4b62-af04-df1bc307ecda" containerID="dab454577d84c14de1f49bcb15ce8a8df95d917d48aabae1f823df4a718b83ad" exitCode=0 Nov 03 22:37:01.424011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:01.423891 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" event={"ID":"80f18706-2830-4b62-af04-df1bc307ecda","Type":"ContainerDied","Data":"dab454577d84c14de1f49bcb15ce8a8df95d917d48aabae1f823df4a718b83ad"} Nov 03 22:37:01.789189 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:01.788890 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:37:02.017572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.017348 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-certs\") pod \"80f18706-2830-4b62-af04-df1bc307ecda\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " Nov 03 22:37:02.017572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.017429 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-envoy\") pod \"80f18706-2830-4b62-af04-df1bc307ecda\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " Nov 03 22:37:02.017572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.017478 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f18706-2830-4b62-af04-df1bc307ecda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f18706-2830-4b62-af04-df1bc307ecda-istiod-ca-cert\") pod \"80f18706-2830-4b62-af04-df1bc307ecda\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " Nov 03 22:37:02.017882 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.017627 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-istio-token\" (UniqueName: \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-istio-token\") pod \"80f18706-2830-4b62-af04-df1bc307ecda\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " Nov 03 22:37:02.017882 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.017676 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-kube-api-access-8w4sq\" (UniqueName: \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-kube-api-access-8w4sq\") pod \"80f18706-2830-4b62-af04-df1bc307ecda\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " Nov 03 22:37:02.017882 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.017717 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-certs" pod "80f18706-2830-4b62-af04-df1bc307ecda" (UID: "80f18706-2830-4b62-af04-df1bc307ecda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:02.017882 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.017758 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-credential-socket\") pod \"80f18706-2830-4b62-af04-df1bc307ecda\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " Nov 03 22:37:02.017882 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.017852 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-socket\") pod \"80f18706-2830-4b62-af04-df1bc307ecda\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " Nov 03 22:37:02.018180 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.017943 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-data\") pod \"80f18706-2830-4b62-af04-df1bc307ecda\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " Nov 03 22:37:02.018180 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.018013 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80f18706-2830-4b62-af04-df1bc307ecda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f18706-2830-4b62-af04-df1bc307ecda-istio-podinfo\") pod \"80f18706-2830-4b62-af04-df1bc307ecda\" (UID: \"80f18706-2830-4b62-af04-df1bc307ecda\") " Nov 03 22:37:02.018918 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.018602 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-data" pod "80f18706-2830-4b62-af04-df1bc307ecda" (UID: "80f18706-2830-4b62-af04-df1bc307ecda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:02.018918 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.018720 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:02.018918 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.018740 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:02.030852 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.025990 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-credential-socket" pod "80f18706-2830-4b62-af04-df1bc307ecda" (UID: "80f18706-2830-4b62-af04-df1bc307ecda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:02.031216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.031081 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-socket" pod "80f18706-2830-4b62-af04-df1bc307ecda" (UID: "80f18706-2830-4b62-af04-df1bc307ecda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:02.031599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.031574 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f18706-2830-4b62-af04-df1bc307ecda-istiod-ca-cert" pod "80f18706-2830-4b62-af04-df1bc307ecda" (UID: "80f18706-2830-4b62-af04-df1bc307ecda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:02.054080 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.054024 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-istio-token" pod "80f18706-2830-4b62-af04-df1bc307ecda" (UID: "80f18706-2830-4b62-af04-df1bc307ecda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:02.054340 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.054302 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80f18706-2830-4b62-af04-df1bc307ecda-istio-podinfo" pod "80f18706-2830-4b62-af04-df1bc307ecda" (UID: "80f18706-2830-4b62-af04-df1bc307ecda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:37:02.116567 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.114674 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-kube-api-access-8w4sq" pod "80f18706-2830-4b62-af04-df1bc307ecda" (UID: "80f18706-2830-4b62-af04-df1bc307ecda"). InnerVolumeSpecName "kube-api-access-8w4sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:02.117892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.117852 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-envoy" pod "80f18706-2830-4b62-af04-df1bc307ecda" (UID: "80f18706-2830-4b62-af04-df1bc307ecda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:02.120554 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.120525 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f18706-2830-4b62-af04-df1bc307ecda-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:02.120757 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.120731 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:02.120757 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.120757 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f18706-2830-4b62-af04-df1bc307ecda-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:02.120926 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.120788 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:02.120926 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.120804 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w4sq\" (UniqueName: \"kubernetes.io/projected/80f18706-2830-4b62-af04-df1bc307ecda-kube-api-access-8w4sq\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:02.120926 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.120820 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:02.120926 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.120834 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f18706-2830-4b62-af04-df1bc307ecda-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:02.432143 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.432107 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" event={"ID":"80f18706-2830-4b62-af04-df1bc307ecda","Type":"ContainerDied","Data":"20ca115a25c571cce9009899f72ecf5ed900a57e7812b42e95fc7ff720f6c298"} Nov 03 22:37:02.432143 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.432130 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz" Nov 03 22:37:02.432377 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.432169 6234 scope.go:122] "RemoveContainer" containerID="dab454577d84c14de1f49bcb15ce8a8df95d917d48aabae1f823df4a718b83ad" Nov 03 22:37:02.596959 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.596861 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz"] Nov 03 22:37:02.635838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.635797 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-p6wkz"] Nov 03 22:37:02.710376 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:02.710339 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f18706-2830-4b62-af04-df1bc307ecda" path="/var/lib/kubelet/pods/80f18706-2830-4b62-af04-df1bc307ecda/volumes" Nov 03 22:37:28.998433 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:37:28.998396 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:37:42.283802 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:37:42.283761 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:37:59.000344 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:37:59.000280 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:38:27.339704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:27.339583 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h"] Nov 03 22:38:27.340355 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:27.339814 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" containerID="cri-o://a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b" gracePeriod=30 Nov 03 22:38:28.996330 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:38:28.996292 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:38:32.593444 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.593388 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:38:32.675477 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.675446 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:38:32.715643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.715612 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47fc9be-c015-4395-91fb-94ecfe8ea648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47fc9be-c015-4395-91fb-94ecfe8ea648-istiod-ca-cert\") pod \"f47fc9be-c015-4395-91fb-94ecfe8ea648\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " Nov 03 22:38:32.715837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.715675 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-socket\") pod \"f47fc9be-c015-4395-91fb-94ecfe8ea648\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " Nov 03 22:38:32.715837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.715705 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-token\" (UniqueName: \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-token\") pod \"f47fc9be-c015-4395-91fb-94ecfe8ea648\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " Nov 03 22:38:32.715837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.715731 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-podinfo\") pod \"f47fc9be-c015-4395-91fb-94ecfe8ea648\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " Nov 03 22:38:32.715837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.715793 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-certs\") pod \"f47fc9be-c015-4395-91fb-94ecfe8ea648\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " Nov 03 22:38:32.716057 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.716027 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47fc9be-c015-4395-91fb-94ecfe8ea648-istiod-ca-cert" pod "f47fc9be-c015-4395-91fb-94ecfe8ea648" (UID: "f47fc9be-c015-4395-91fb-94ecfe8ea648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:32.716057 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.716041 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-kube-api-access-q6vh5\" (UniqueName: \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-kube-api-access-q6vh5\") pod \"f47fc9be-c015-4395-91fb-94ecfe8ea648\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " Nov 03 22:38:32.716168 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.716101 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-credential-socket\") pod \"f47fc9be-c015-4395-91fb-94ecfe8ea648\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " Nov 03 22:38:32.716168 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.716154 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-envoy\") pod \"f47fc9be-c015-4395-91fb-94ecfe8ea648\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " Nov 03 22:38:32.716285 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.716190 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-data\") pod \"f47fc9be-c015-4395-91fb-94ecfe8ea648\" (UID: \"f47fc9be-c015-4395-91fb-94ecfe8ea648\") " Nov 03 22:38:32.716399 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.716379 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-credential-socket" pod "f47fc9be-c015-4395-91fb-94ecfe8ea648" (UID: "f47fc9be-c015-4395-91fb-94ecfe8ea648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:32.716463 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.716401 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-certs" pod "f47fc9be-c015-4395-91fb-94ecfe8ea648" (UID: "f47fc9be-c015-4395-91fb-94ecfe8ea648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:32.716687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.716663 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-data" pod "f47fc9be-c015-4395-91fb-94ecfe8ea648" (UID: "f47fc9be-c015-4395-91fb-94ecfe8ea648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:32.716773 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.716744 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-socket" pod "f47fc9be-c015-4395-91fb-94ecfe8ea648" (UID: "f47fc9be-c015-4395-91fb-94ecfe8ea648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:32.717061 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.717038 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.717160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.717066 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.717160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.717080 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47fc9be-c015-4395-91fb-94ecfe8ea648-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.717160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.717094 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.717160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.717107 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.718925 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.718898 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-envoy" pod "f47fc9be-c015-4395-91fb-94ecfe8ea648" (UID: "f47fc9be-c015-4395-91fb-94ecfe8ea648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:32.719171 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.719149 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-token" pod "f47fc9be-c015-4395-91fb-94ecfe8ea648" (UID: "f47fc9be-c015-4395-91fb-94ecfe8ea648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:32.719476 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.719446 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-kube-api-access-q6vh5" pod "f47fc9be-c015-4395-91fb-94ecfe8ea648" (UID: "f47fc9be-c015-4395-91fb-94ecfe8ea648"). InnerVolumeSpecName "kube-api-access-q6vh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:32.719603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.719480 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-podinfo" pod "f47fc9be-c015-4395-91fb-94ecfe8ea648" (UID: "f47fc9be-c015-4395-91fb-94ecfe8ea648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:32.823518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.821240 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.823518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.821276 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.823518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.821307 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6vh5\" (UniqueName: \"kubernetes.io/projected/f47fc9be-c015-4395-91fb-94ecfe8ea648-kube-api-access-q6vh5\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.823518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:32.821324 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47fc9be-c015-4395-91fb-94ecfe8ea648-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.110308 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:33.110276 6234 generic.go:356] "Generic (PLEG): container finished" podID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerID="a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b" exitCode=0 Nov 03 22:38:33.110488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:33.110343 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" Nov 03 22:38:33.110488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:33.110369 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" event={"ID":"f47fc9be-c015-4395-91fb-94ecfe8ea648","Type":"ContainerDied","Data":"a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b"} Nov 03 22:38:33.110488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:33.110433 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h" event={"ID":"f47fc9be-c015-4395-91fb-94ecfe8ea648","Type":"ContainerDied","Data":"5767926f398cee95e1089524f2bbec252b58838ec6cf7cb64e41e4db9c297637"} Nov 03 22:38:33.110488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:33.110460 6234 scope.go:122] "RemoveContainer" containerID="a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b" Nov 03 22:38:33.122792 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:33.122771 6234 scope.go:122] "RemoveContainer" containerID="a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b" Nov 03 22:38:33.123136 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:38:33.123111 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b\": container with ID starting with a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b not found: ID does not exist" containerID="a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b" Nov 03 22:38:33.123251 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:33.123146 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b"} err="failed to get container status \"a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b\": rpc error: code = NotFound desc = could not find container \"a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b\": container with ID starting with a498b58f4e638179fae1ca45af0f6bc1f805ffa1bbefc391cc6341bdfd08866b not found: ID does not exist" Nov 03 22:38:33.270410 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:33.270367 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h"] Nov 03 22:38:33.274629 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:33.274583 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-vz47h"] Nov 03 22:38:34.706075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:34.706044 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" path="/var/lib/kubelet/pods/f47fc9be-c015-4395-91fb-94ecfe8ea648/volumes" Nov 03 22:38:41.480626 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:41.480584 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x"] Nov 03 22:38:41.481145 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:41.480896 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" containerID="cri-o://ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd" gracePeriod=30 Nov 03 22:38:42.110197 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:42.110154 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b"] Nov 03 22:38:42.110421 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:42.110381 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" containerID="cri-o://e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce" gracePeriod=30 Nov 03 22:38:42.403778 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:42.403696 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm"] Nov 03 22:38:42.403979 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:42.403921 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" containerID="cri-o://3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb" gracePeriod=30 Nov 03 22:38:46.750076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.750012 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:38:46.928291 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928255 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-certs\") pod \"a4d19de6-9f35-42a3-86a2-bfe70247b928\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " Nov 03 22:38:46.928482 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928311 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-podinfo\") pod \"a4d19de6-9f35-42a3-86a2-bfe70247b928\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " Nov 03 22:38:46.928482 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928409 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-credential-socket\") pod \"a4d19de6-9f35-42a3-86a2-bfe70247b928\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " Nov 03 22:38:46.928482 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928438 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-token\" (UniqueName: \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-token\") pod \"a4d19de6-9f35-42a3-86a2-bfe70247b928\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " Nov 03 22:38:46.928698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928514 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-certs" pod "a4d19de6-9f35-42a3-86a2-bfe70247b928" (UID: "a4d19de6-9f35-42a3-86a2-bfe70247b928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:46.928698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928565 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d19de6-9f35-42a3-86a2-bfe70247b928-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d19de6-9f35-42a3-86a2-bfe70247b928-istiod-ca-cert\") pod \"a4d19de6-9f35-42a3-86a2-bfe70247b928\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " Nov 03 22:38:46.928698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928687 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-credential-socket" pod "a4d19de6-9f35-42a3-86a2-bfe70247b928" (UID: "a4d19de6-9f35-42a3-86a2-bfe70247b928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:46.928857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928778 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-envoy\") pod \"a4d19de6-9f35-42a3-86a2-bfe70247b928\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " Nov 03 22:38:46.928857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928827 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-socket\") pod \"a4d19de6-9f35-42a3-86a2-bfe70247b928\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " Nov 03 22:38:46.928952 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928877 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-data\") pod \"a4d19de6-9f35-42a3-86a2-bfe70247b928\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " Nov 03 22:38:46.928952 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.928910 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-kube-api-access-tcf24\" (UniqueName: \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-kube-api-access-tcf24\") pod \"a4d19de6-9f35-42a3-86a2-bfe70247b928\" (UID: \"a4d19de6-9f35-42a3-86a2-bfe70247b928\") " Nov 03 22:38:46.929087 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.929067 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d19de6-9f35-42a3-86a2-bfe70247b928-istiod-ca-cert" pod "a4d19de6-9f35-42a3-86a2-bfe70247b928" (UID: "a4d19de6-9f35-42a3-86a2-bfe70247b928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:46.929323 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.929303 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-socket" pod "a4d19de6-9f35-42a3-86a2-bfe70247b928" (UID: "a4d19de6-9f35-42a3-86a2-bfe70247b928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:46.929620 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.929592 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.929620 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.929615 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.929821 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.929629 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.929821 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.929642 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d19de6-9f35-42a3-86a2-bfe70247b928-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:46.929821 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.929757 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-data" pod "a4d19de6-9f35-42a3-86a2-bfe70247b928" (UID: "a4d19de6-9f35-42a3-86a2-bfe70247b928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:46.931994 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.931923 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-token" pod "a4d19de6-9f35-42a3-86a2-bfe70247b928" (UID: "a4d19de6-9f35-42a3-86a2-bfe70247b928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:46.935263 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.935238 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-kube-api-access-tcf24" pod "a4d19de6-9f35-42a3-86a2-bfe70247b928" (UID: "a4d19de6-9f35-42a3-86a2-bfe70247b928"). InnerVolumeSpecName "kube-api-access-tcf24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:46.935798 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.935775 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-envoy" pod "a4d19de6-9f35-42a3-86a2-bfe70247b928" (UID: "a4d19de6-9f35-42a3-86a2-bfe70247b928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:46.936143 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:46.936111 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-podinfo" pod "a4d19de6-9f35-42a3-86a2-bfe70247b928" (UID: "a4d19de6-9f35-42a3-86a2-bfe70247b928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:47.002030 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.001960 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:38:47.002030 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.002012 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:38:47.031549 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.031518 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.031549 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.031547 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.031757 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.031558 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcf24\" (UniqueName: \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-kube-api-access-tcf24\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.031757 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.031567 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.031757 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.031576 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d19de6-9f35-42a3-86a2-bfe70247b928-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.209555 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.209519 6234 generic.go:356] "Generic (PLEG): container finished" podID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerID="ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd" exitCode=0 Nov 03 22:38:47.209702 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.209577 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" Nov 03 22:38:47.209702 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.209612 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" event={"ID":"a4d19de6-9f35-42a3-86a2-bfe70247b928","Type":"ContainerDied","Data":"ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd"} Nov 03 22:38:47.209702 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.209678 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x" event={"ID":"a4d19de6-9f35-42a3-86a2-bfe70247b928","Type":"ContainerDied","Data":"ac6bafc9f836bb2cc6bd7ff3781beab5ec5bafa82cdaad173cae4a325fe44fc9"} Nov 03 22:38:47.209859 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.209709 6234 scope.go:122] "RemoveContainer" containerID="ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd" Nov 03 22:38:47.223811 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.223787 6234 scope.go:122] "RemoveContainer" containerID="ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd" Nov 03 22:38:47.224603 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:38:47.224579 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd\": container with ID starting with ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd not found: ID does not exist" containerID="ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd" Nov 03 22:38:47.224728 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.224616 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd"} err="failed to get container status \"ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd\": rpc error: code = NotFound desc = could not find container \"ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd\": container with ID starting with ba3c9f32f8498ade5a461995fd19dd3b5de036a67ed55e9abd1eec57e74a3ccd not found: ID does not exist" Nov 03 22:38:47.373564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.368229 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x"] Nov 03 22:38:47.373564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.373357 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-gjl2x"] Nov 03 22:38:47.392876 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.392849 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:38:47.540721 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.540693 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-envoy\") pod \"a8219d9c-58a1-4588-a298-7066d5df5d1f\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " Nov 03 22:38:47.540887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.540759 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-kube-api-access-qst4t\" (UniqueName: \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-kube-api-access-qst4t\") pod \"a8219d9c-58a1-4588-a298-7066d5df5d1f\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " Nov 03 22:38:47.540887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.540866 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8219d9c-58a1-4588-a298-7066d5df5d1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8219d9c-58a1-4588-a298-7066d5df5d1f-istiod-ca-cert\") pod \"a8219d9c-58a1-4588-a298-7066d5df5d1f\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " Nov 03 22:38:47.541008 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.540907 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-credential-socket\") pod \"a8219d9c-58a1-4588-a298-7066d5df5d1f\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " Nov 03 22:38:47.541008 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.540998 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-certs\") pod \"a8219d9c-58a1-4588-a298-7066d5df5d1f\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " Nov 03 22:38:47.541110 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.541082 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-data\") pod \"a8219d9c-58a1-4588-a298-7066d5df5d1f\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " Nov 03 22:38:47.541599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.541200 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-podinfo\") pod \"a8219d9c-58a1-4588-a298-7066d5df5d1f\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " Nov 03 22:38:47.541599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.541248 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-token\" (UniqueName: \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-token\") pod \"a8219d9c-58a1-4588-a298-7066d5df5d1f\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " Nov 03 22:38:47.541599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.541286 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-socket\") pod \"a8219d9c-58a1-4588-a298-7066d5df5d1f\" (UID: \"a8219d9c-58a1-4588-a298-7066d5df5d1f\") " Nov 03 22:38:47.541599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.541420 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-certs" pod "a8219d9c-58a1-4588-a298-7066d5df5d1f" (UID: "a8219d9c-58a1-4588-a298-7066d5df5d1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.541599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.541490 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8219d9c-58a1-4588-a298-7066d5df5d1f-istiod-ca-cert" pod "a8219d9c-58a1-4588-a298-7066d5df5d1f" (UID: "a8219d9c-58a1-4588-a298-7066d5df5d1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:47.541965 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.541660 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-credential-socket" pod "a8219d9c-58a1-4588-a298-7066d5df5d1f" (UID: "a8219d9c-58a1-4588-a298-7066d5df5d1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.541965 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.541954 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-data" pod "a8219d9c-58a1-4588-a298-7066d5df5d1f" (UID: "a8219d9c-58a1-4588-a298-7066d5df5d1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.542084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.541981 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8219d9c-58a1-4588-a298-7066d5df5d1f-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.542084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.541998 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.542084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.542006 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.542254 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.542108 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-socket" pod "a8219d9c-58a1-4588-a298-7066d5df5d1f" (UID: "a8219d9c-58a1-4588-a298-7066d5df5d1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.544198 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.544176 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-podinfo" pod "a8219d9c-58a1-4588-a298-7066d5df5d1f" (UID: "a8219d9c-58a1-4588-a298-7066d5df5d1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:47.544308 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.544229 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-token" pod "a8219d9c-58a1-4588-a298-7066d5df5d1f" (UID: "a8219d9c-58a1-4588-a298-7066d5df5d1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.544379 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.544298 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-envoy" pod "a8219d9c-58a1-4588-a298-7066d5df5d1f" (UID: "a8219d9c-58a1-4588-a298-7066d5df5d1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.544916 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.544893 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-kube-api-access-qst4t" pod "a8219d9c-58a1-4588-a298-7066d5df5d1f" (UID: "a8219d9c-58a1-4588-a298-7066d5df5d1f"). InnerVolumeSpecName "kube-api-access-qst4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.643292 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.643259 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.643292 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.643288 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.643292 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.643298 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.643615 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.643308 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.643615 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.643318 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qst4t\" (UniqueName: \"kubernetes.io/projected/a8219d9c-58a1-4588-a298-7066d5df5d1f-kube-api-access-qst4t\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.643615 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.643328 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8219d9c-58a1-4588-a298-7066d5df5d1f-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.689112 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.689086 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:38:47.845241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.845159 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-kube-api-access-gx4dm\" (UniqueName: \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-kube-api-access-gx4dm\") pod \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " Nov 03 22:38:47.845241 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.845196 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-podinfo\") pod \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " Nov 03 22:38:47.845836 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.845299 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-socket\") pod \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " Nov 03 22:38:47.845836 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.845432 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-credential-socket\") pod \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " Nov 03 22:38:47.845836 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.845506 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istiod-ca-cert\") pod \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " Nov 03 22:38:47.845836 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.845563 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-envoy\") pod \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " Nov 03 22:38:47.845836 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.845610 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-data\") pod \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " Nov 03 22:38:47.845836 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.845643 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-token\" (UniqueName: \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-token\") pod \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " Nov 03 22:38:47.845836 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.845704 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-socket" pod "691ef725-0a35-48a2-a393-56ddf5b1b3f6" (UID: "691ef725-0a35-48a2-a393-56ddf5b1b3f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.845836 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.845740 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-certs\") pod \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\" (UID: \"691ef725-0a35-48a2-a393-56ddf5b1b3f6\") " Nov 03 22:38:47.846410 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.846378 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.846659 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.846615 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-credential-socket" pod "691ef725-0a35-48a2-a393-56ddf5b1b3f6" (UID: "691ef725-0a35-48a2-a393-56ddf5b1b3f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.846784 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.846759 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istiod-ca-cert" pod "691ef725-0a35-48a2-a393-56ddf5b1b3f6" (UID: "691ef725-0a35-48a2-a393-56ddf5b1b3f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:47.847043 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.847022 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-certs" pod "691ef725-0a35-48a2-a393-56ddf5b1b3f6" (UID: "691ef725-0a35-48a2-a393-56ddf5b1b3f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.847262 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.847234 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-data" pod "691ef725-0a35-48a2-a393-56ddf5b1b3f6" (UID: "691ef725-0a35-48a2-a393-56ddf5b1b3f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.848177 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.848147 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-kube-api-access-gx4dm" pod "691ef725-0a35-48a2-a393-56ddf5b1b3f6" (UID: "691ef725-0a35-48a2-a393-56ddf5b1b3f6"). InnerVolumeSpecName "kube-api-access-gx4dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.848274 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.848214 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-podinfo" pod "691ef725-0a35-48a2-a393-56ddf5b1b3f6" (UID: "691ef725-0a35-48a2-a393-56ddf5b1b3f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:47.848315 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.848297 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-envoy" pod "691ef725-0a35-48a2-a393-56ddf5b1b3f6" (UID: "691ef725-0a35-48a2-a393-56ddf5b1b3f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.848922 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.848903 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-token" pod "691ef725-0a35-48a2-a393-56ddf5b1b3f6" (UID: "691ef725-0a35-48a2-a393-56ddf5b1b3f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.947941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.947904 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.947941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.947928 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.947941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.947937 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.948232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.947958 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.948232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.947967 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.948232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.947976 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691ef725-0a35-48a2-a393-56ddf5b1b3f6-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.948232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.947985 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx4dm\" (UniqueName: \"kubernetes.io/projected/691ef725-0a35-48a2-a393-56ddf5b1b3f6-kube-api-access-gx4dm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.948232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:47.947996 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691ef725-0a35-48a2-a393-56ddf5b1b3f6-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:48.219183 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.219149 6234 generic.go:356] "Generic (PLEG): container finished" podID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerID="3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb" exitCode=0 Nov 03 22:38:48.219437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.219209 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" Nov 03 22:38:48.219437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.219238 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" event={"ID":"691ef725-0a35-48a2-a393-56ddf5b1b3f6","Type":"ContainerDied","Data":"3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb"} Nov 03 22:38:48.219437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.219295 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm" event={"ID":"691ef725-0a35-48a2-a393-56ddf5b1b3f6","Type":"ContainerDied","Data":"0d0f9938f407b4161a9559f99572a3cbd7f197adf245b982e753917c60cc4169"} Nov 03 22:38:48.219437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.219320 6234 scope.go:122] "RemoveContainer" containerID="3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb" Nov 03 22:38:48.221687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.221560 6234 generic.go:356] "Generic (PLEG): container finished" podID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerID="e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce" exitCode=0 Nov 03 22:38:48.221687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.221611 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" Nov 03 22:38:48.221687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.221613 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" event={"ID":"a8219d9c-58a1-4588-a298-7066d5df5d1f","Type":"ContainerDied","Data":"e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce"} Nov 03 22:38:48.221687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.221657 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b" event={"ID":"a8219d9c-58a1-4588-a298-7066d5df5d1f","Type":"ContainerDied","Data":"5f096d939981e213019ae71a54703b3f96ad1b8762d853188dbdb0c61030275e"} Nov 03 22:38:48.230996 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.230977 6234 scope.go:122] "RemoveContainer" containerID="3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb" Nov 03 22:38:48.231312 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:38:48.231289 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb\": container with ID starting with 3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb not found: ID does not exist" containerID="3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb" Nov 03 22:38:48.231398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.231318 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb"} err="failed to get container status \"3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb\": rpc error: code = NotFound desc = could not find container \"3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb\": container with ID starting with 3cae2fd47f35b5b66f474bf1b016227b42ad678f1499c679dac37704ab9e06fb not found: ID does not exist" Nov 03 22:38:48.231398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.231335 6234 scope.go:122] "RemoveContainer" containerID="e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce" Nov 03 22:38:48.240262 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.240243 6234 scope.go:122] "RemoveContainer" containerID="e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce" Nov 03 22:38:48.241150 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:38:48.241126 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce\": container with ID starting with e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce not found: ID does not exist" containerID="e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce" Nov 03 22:38:48.241237 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.241155 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce"} err="failed to get container status \"e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce\": rpc error: code = NotFound desc = could not find container \"e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce\": container with ID starting with e72f4f7b88dbafc3d32f4d2b3c805e2ad9336c6c7a5815d9f9225fedfad174ce not found: ID does not exist" Nov 03 22:38:48.373450 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.373420 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm"] Nov 03 22:38:48.378688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.378661 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d26lm"] Nov 03 22:38:48.480647 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.480575 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b"] Nov 03 22:38:48.487231 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.487205 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-kf56b"] Nov 03 22:38:48.706369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.706334 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" path="/var/lib/kubelet/pods/691ef725-0a35-48a2-a393-56ddf5b1b3f6/volumes" Nov 03 22:38:48.706886 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.706871 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" path="/var/lib/kubelet/pods/a4d19de6-9f35-42a3-86a2-bfe70247b928/volumes" Nov 03 22:38:48.707302 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:48.707289 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" path="/var/lib/kubelet/pods/a8219d9c-58a1-4588-a298-7066d5df5d1f/volumes" Nov 03 22:38:55.978098 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:55.978056 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh"] Nov 03 22:38:55.978617 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:38:55.978323 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" containerID="cri-o://5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11" gracePeriod=30 Nov 03 22:38:59.004049 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:38:59.004011 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:39:01.262801 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.262735 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:39:01.309238 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.309198 6234 generic.go:356] "Generic (PLEG): container finished" podID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerID="5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11" exitCode=0 Nov 03 22:39:01.309398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.309251 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" Nov 03 22:39:01.309398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.309283 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" event={"ID":"6d426a10-fbce-4d15-bb78-cb3d3b40e811","Type":"ContainerDied","Data":"5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11"} Nov 03 22:39:01.309398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.309363 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh" event={"ID":"6d426a10-fbce-4d15-bb78-cb3d3b40e811","Type":"ContainerDied","Data":"6f8480b4b04d2c561b16a288afb33ade4cca7480764b76c6f76ef618236ba3c3"} Nov 03 22:39:01.309398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.309391 6234 scope.go:122] "RemoveContainer" containerID="5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11" Nov 03 22:39:01.318956 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.318939 6234 scope.go:122] "RemoveContainer" containerID="5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11" Nov 03 22:39:01.319245 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:39:01.319227 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11\": container with ID starting with 5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11 not found: ID does not exist" containerID="5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11" Nov 03 22:39:01.319293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.319255 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11"} err="failed to get container status \"5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11\": rpc error: code = NotFound desc = could not find container \"5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11\": container with ID starting with 5abf8276c1b7cff6a7fa65e597be617bfc6bef71a8582a608f079a9b6b6f8c11 not found: ID does not exist" Nov 03 22:39:01.327690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.327671 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-envoy\") pod \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " Nov 03 22:39:01.327778 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.327715 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-token\" (UniqueName: \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-token\") pod \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " Nov 03 22:39:01.327844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.327803 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-credential-socket\") pod \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " Nov 03 22:39:01.327899 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.327847 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-data\") pod \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " Nov 03 22:39:01.327899 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.327878 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-socket\") pod \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " Nov 03 22:39:01.328006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.327909 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-podinfo\") pod \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " Nov 03 22:39:01.328006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.327957 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istiod-ca-cert\") pod \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " Nov 03 22:39:01.328121 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328028 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-kube-api-access-gbd7w\" (UniqueName: \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-kube-api-access-gbd7w\") pod \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " Nov 03 22:39:01.328175 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328115 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-credential-socket" pod "6d426a10-fbce-4d15-bb78-cb3d3b40e811" (UID: "6d426a10-fbce-4d15-bb78-cb3d3b40e811"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.328175 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328129 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-certs\") pod \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\" (UID: \"6d426a10-fbce-4d15-bb78-cb3d3b40e811\") " Nov 03 22:39:01.328175 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328131 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-data" pod "6d426a10-fbce-4d15-bb78-cb3d3b40e811" (UID: "6d426a10-fbce-4d15-bb78-cb3d3b40e811"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.328429 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328404 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-certs" pod "6d426a10-fbce-4d15-bb78-cb3d3b40e811" (UID: "6d426a10-fbce-4d15-bb78-cb3d3b40e811"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.328523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328483 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istiod-ca-cert" pod "6d426a10-fbce-4d15-bb78-cb3d3b40e811" (UID: "6d426a10-fbce-4d15-bb78-cb3d3b40e811"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:01.328772 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328745 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-socket" pod "6d426a10-fbce-4d15-bb78-cb3d3b40e811" (UID: "6d426a10-fbce-4d15-bb78-cb3d3b40e811"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.328897 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328884 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.328976 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328906 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.328976 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328920 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.328976 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328933 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.328976 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.328947 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.329958 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.329928 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-envoy" pod "6d426a10-fbce-4d15-bb78-cb3d3b40e811" (UID: "6d426a10-fbce-4d15-bb78-cb3d3b40e811"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.330120 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.330102 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-token" pod "6d426a10-fbce-4d15-bb78-cb3d3b40e811" (UID: "6d426a10-fbce-4d15-bb78-cb3d3b40e811"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:01.330202 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.330158 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-kube-api-access-gbd7w" pod "6d426a10-fbce-4d15-bb78-cb3d3b40e811" (UID: "6d426a10-fbce-4d15-bb78-cb3d3b40e811"). InnerVolumeSpecName "kube-api-access-gbd7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:01.330321 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.330300 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-podinfo" pod "6d426a10-fbce-4d15-bb78-cb3d3b40e811" (UID: "6d426a10-fbce-4d15-bb78-cb3d3b40e811"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:39:01.430583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.430552 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.430583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.430580 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.430773 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.430590 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d426a10-fbce-4d15-bb78-cb3d3b40e811-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.430773 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.430601 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbd7w\" (UniqueName: \"kubernetes.io/projected/6d426a10-fbce-4d15-bb78-cb3d3b40e811-kube-api-access-gbd7w\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.728308 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.728270 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh"] Nov 03 22:39:01.731654 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:01.731622 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7gcgh"] Nov 03 22:39:02.706171 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:02.706138 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" path="/var/lib/kubelet/pods/6d426a10-fbce-4d15-bb78-cb3d3b40e811/volumes" Nov 03 22:39:29.002126 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:39:29.002075 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:39:32.840987 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.840954 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw"] Nov 03 22:39:32.841623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841606 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" Nov 03 22:39:32.841688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841624 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" Nov 03 22:39:32.841688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841635 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" Nov 03 22:39:32.841688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841643 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" Nov 03 22:39:32.841688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841656 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" Nov 03 22:39:32.841688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841661 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" Nov 03 22:39:32.841688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841671 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" Nov 03 22:39:32.841688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841676 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" Nov 03 22:39:32.841688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841690 6234 cpu_manager.go:414] "containerMap: removing container" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" Nov 03 22:39:32.841964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841694 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" Nov 03 22:39:32.841964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841701 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerName="istio-proxy" Nov 03 22:39:32.841964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841706 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerName="istio-proxy" Nov 03 22:39:32.841964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841748 6234 cpu_manager.go:414] "containerMap: removing container" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" Nov 03 22:39:32.841964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841755 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" Nov 03 22:39:32.841964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841766 6234 cpu_manager.go:414] "containerMap: removing container" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" Nov 03 22:39:32.841964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841772 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" Nov 03 22:39:32.841964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841828 6234 cpu_manager.go:414] "containerMap: removing container" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" Nov 03 22:39:32.841964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.841833 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" Nov 03 22:39:32.842280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.842020 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="691ef725-0a35-48a2-a393-56ddf5b1b3f6" containerName="istio-proxy" Nov 03 22:39:32.842280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.842037 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f18706-2830-4b62-af04-df1bc307ecda" containerName="istio-proxy" Nov 03 22:39:32.842280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.842056 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47fc9be-c015-4395-91fb-94ecfe8ea648" containerName="istio-proxy" Nov 03 22:39:32.842280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.842068 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8219d9c-58a1-4588-a298-7066d5df5d1f" containerName="istio-proxy" Nov 03 22:39:32.842280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.842076 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d19de6-9f35-42a3-86a2-bfe70247b928" containerName="istio-proxy" Nov 03 22:39:32.842280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.842083 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6861036d-82cd-413b-88c2-df43b2b7a613" containerName="istio-proxy" Nov 03 22:39:32.842280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.842091 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d426a10-fbce-4d15-bb78-cb3d3b40e811" containerName="istio-proxy" Nov 03 22:39:32.842280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.842098 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fad8571-4357-4318-8f8a-6515d4211a9a" containerName="istio-proxy" Nov 03 22:39:32.842280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.842105 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d34efe6-93ed-4dc1-ba14-896977c661b2" containerName="istio-proxy" Nov 03 22:39:32.847040 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.847021 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:32.896339 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.896300 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-345da9-openshift-default-dockercfg-ln6f7\"" Nov 03 22:39:32.947440 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.947406 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw"] Nov 03 22:39:32.957290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.957257 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-certs\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:32.957460 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.957319 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-data\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:32.957460 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.957412 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-envoy\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:32.957601 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.957556 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r94n\" (UniqueName: \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-kube-api-access-2r94n\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:32.957660 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.957597 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-podinfo\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:32.957660 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.957646 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-credential-socket\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:32.957767 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.957736 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-socket\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:32.957822 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.957785 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-token\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:32.957875 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:32.957820 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ab3852-4c26-42b9-9927-926bb9a00e79-istiod-ca-cert\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060053 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060005 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-certs\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060069 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-data\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060101 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-envoy\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060194 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r94n\" (UniqueName: \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-kube-api-access-2r94n\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060223 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-podinfo\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060303 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-credential-socket\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060483 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-certs\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060568 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-data\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060709 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-credential-socket\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060812 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060762 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-socket\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060876 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060837 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-socket\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.060876 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060865 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-token\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.061076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.060904 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ab3852-4c26-42b9-9927-926bb9a00e79-istiod-ca-cert\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.062458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.061472 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ab3852-4c26-42b9-9927-926bb9a00e79-istiod-ca-cert\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.063128 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.063100 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-envoy\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.063128 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.063123 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-podinfo\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.114704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.114637 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-token\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.114833 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.114743 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r94n\" (UniqueName: \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-kube-api-access-2r94n\") pod \"e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.160743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.160707 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:33.605785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.605747 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw"] Nov 03 22:39:33.611927 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:39:33.611895 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ab3852_4c26_42b9_9927_926bb9a00e79.slice/crio-77672c4c7c255291bf56c9b73ed646eed6d579b555f99ebc564b94e4e9ef339e WatchSource:0}: Error finding container 77672c4c7c255291bf56c9b73ed646eed6d579b555f99ebc564b94e4e9ef339e: Status 404 returned error can't find the container with id 77672c4c7c255291bf56c9b73ed646eed6d579b555f99ebc564b94e4e9ef339e Nov 03 22:39:33.614808 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.614771 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:33.614912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.614843 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:33.614912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.614871 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:33.746684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:33.746643 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:39:34.529316 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:34.529277 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" event={"ID":"c6ab3852-4c26-42b9-9927-926bb9a00e79","Type":"ContainerStarted","Data":"19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea"} Nov 03 22:39:34.529849 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:34.529332 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" event={"ID":"c6ab3852-4c26-42b9-9927-926bb9a00e79","Type":"ContainerStarted","Data":"77672c4c7c255291bf56c9b73ed646eed6d579b555f99ebc564b94e4e9ef339e"} Nov 03 22:39:34.735927 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:34.735872 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" podStartSLOduration=2.735854423 podStartE2EDuration="2.735854423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:34.687836617 +0000 UTC m=+1894.100022635" watchObservedRunningTime="2026-11-03 22:39:34.735854423 +0000 UTC m=+1894.148040442" Nov 03 22:39:35.160870 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:35.160838 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:35.167405 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:35.167378 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:35.572676 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:35.572636 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:35.578302 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:35.578260 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:39:39.392183 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.392148 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn"] Nov 03 22:39:39.396730 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.396706 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.445699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.445665 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-2a6801-openshift-default-dockercfg-nqtbg\"" Nov 03 22:39:39.464261 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.464222 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.464452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.464280 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.464452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.464419 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5xrg\" (UniqueName: \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-kube-api-access-q5xrg\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.464452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.464447 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.464711 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.464563 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.464711 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.464597 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.464711 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.464655 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d62724-a2ac-43f0-8f35-9fc114109d79-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.464711 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.464693 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.464913 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.464757 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.496127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.496093 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn"] Nov 03 22:39:39.566065 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566031 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566281 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566103 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566281 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566264 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566402 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566295 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566417 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5xrg\" (UniqueName: \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-kube-api-access-q5xrg\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566417 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566458 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566447 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566635 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566517 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566635 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566545 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566635 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566603 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d62724-a2ac-43f0-8f35-9fc114109d79-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566801 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566711 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566801 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566745 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.566884 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.566851 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.567420 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.567388 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d62724-a2ac-43f0-8f35-9fc114109d79-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.568993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.568959 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.569113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.569079 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.623071 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.623033 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.623326 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.623302 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5xrg\" (UniqueName: \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-kube-api-access-q5xrg\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:39.720452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:39.715933 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:40.154132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:40.154025 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn"] Nov 03 22:39:40.175072 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:40.175026 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:40.175311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:40.175286 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:40.175457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:40.175439 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:40.627517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:40.626876 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" event={"ID":"e7d62724-a2ac-43f0-8f35-9fc114109d79","Type":"ContainerStarted","Data":"bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d"} Nov 03 22:39:40.627517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:40.626949 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" event={"ID":"e7d62724-a2ac-43f0-8f35-9fc114109d79","Type":"ContainerStarted","Data":"252b4f38bc470262b3814ab7a232fd8fc2c9e7ebee087fb0c375bcb9f409b6e7"} Nov 03 22:39:40.718515 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:40.718339 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:40.850801 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:40.846379 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" podStartSLOduration=1.846359046 podStartE2EDuration="1.846359046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:40.845341003 +0000 UTC m=+1900.257527027" watchObservedRunningTime="2026-11-03 22:39:40.846359046 +0000 UTC m=+1900.258545063" Nov 03 22:39:41.761519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:41.760810 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:42.641557 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:42.641517 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:42.642535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:42.642512 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:39:58.997710 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:39:58.997674 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:39:59.755538 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:59.751692 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp"] Nov 03 22:39:59.764254 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:59.763036 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:39:59.813559 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:59.810383 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp"] Nov 03 22:39:59.849834 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:59.849799 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bc88eb47-dockercfg-jxfmd\"" Nov 03 22:39:59.850629 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:59.850579 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bc88eb47\"" Nov 03 22:39:59.929962 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:59.929930 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-plugins-config-volume\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp\" (UID: \"391cd6a8-2698-4fc8-9ede-b4290d03c0b0\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:39:59.930151 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:39:59.930055 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6992\" (UniqueName: \"kubernetes.io/projected/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-kube-api-access-t6992\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp\" (UID: \"391cd6a8-2698-4fc8-9ede-b4290d03c0b0\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:40:00.030960 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:00.030871 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-plugins-config-volume\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp\" (UID: \"391cd6a8-2698-4fc8-9ede-b4290d03c0b0\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:40:00.031319 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:00.030976 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6992\" (UniqueName: \"kubernetes.io/projected/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-kube-api-access-t6992\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp\" (UID: \"391cd6a8-2698-4fc8-9ede-b4290d03c0b0\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:40:00.031578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:00.031561 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-plugins-config-volume\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp\" (UID: \"391cd6a8-2698-4fc8-9ede-b4290d03c0b0\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:40:00.096365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:00.096334 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6992\" (UniqueName: \"kubernetes.io/projected/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-kube-api-access-t6992\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp\" (UID: \"391cd6a8-2698-4fc8-9ede-b4290d03c0b0\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:40:00.390461 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:00.390372 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:40:00.783177 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:00.783144 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp"] Nov 03 22:40:00.787426 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:40:00.787394 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391cd6a8_2698_4fc8_9ede_b4290d03c0b0.slice/crio-32bacd553ce966836c0af5a0f31a0f12f4457f529896bc6780d3ab647eb0ca8e WatchSource:0}: Error finding container 32bacd553ce966836c0af5a0f31a0f12f4457f529896bc6780d3ab647eb0ca8e: Status 404 returned error can't find the container with id 32bacd553ce966836c0af5a0f31a0f12f4457f529896bc6780d3ab647eb0ca8e Nov 03 22:40:01.777599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:01.777558 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" event={"ID":"391cd6a8-2698-4fc8-9ede-b4290d03c0b0","Type":"ContainerStarted","Data":"1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192"} Nov 03 22:40:01.778001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:01.777616 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" event={"ID":"391cd6a8-2698-4fc8-9ede-b4290d03c0b0","Type":"ContainerStarted","Data":"32bacd553ce966836c0af5a0f31a0f12f4457f529896bc6780d3ab647eb0ca8e"} Nov 03 22:40:01.778001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:01.777642 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:40:01.887512 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:01.887442 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" podStartSLOduration=2.887429962 podStartE2EDuration="2.887429962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:01.88420736 +0000 UTC m=+1921.296393374" watchObservedRunningTime="2026-11-03 22:40:01.887429962 +0000 UTC m=+1921.299615978" Nov 03 22:40:12.787358 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:12.787329 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:40:27.407149 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:27.407114 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t"] Nov 03 22:40:27.407673 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:27.407361 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" containerID="cri-o://d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982" gracePeriod=30 Nov 03 22:40:28.997971 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:40:28.997930 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:40:30.474183 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:30.474137 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:40:30.474699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:30.474216 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:40:30.551095 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:30.551058 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:40:32.686154 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.686127 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:40:32.696242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696213 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-socket\") pod \"00367384-f2ca-4817-a170-5720fcb0b1e3\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " Nov 03 22:40:32.696391 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696306 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-data\") pod \"00367384-f2ca-4817-a170-5720fcb0b1e3\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " Nov 03 22:40:32.696391 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696374 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-podinfo\") pod \"00367384-f2ca-4817-a170-5720fcb0b1e3\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " Nov 03 22:40:32.696539 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696427 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-envoy\") pod \"00367384-f2ca-4817-a170-5720fcb0b1e3\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " Nov 03 22:40:32.696603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696532 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-socket" pod "00367384-f2ca-4817-a170-5720fcb0b1e3" (UID: "00367384-f2ca-4817-a170-5720fcb0b1e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:32.696603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696557 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-data" pod "00367384-f2ca-4817-a170-5720fcb0b1e3" (UID: "00367384-f2ca-4817-a170-5720fcb0b1e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:32.696603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696569 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-token\" (UniqueName: \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-token\") pod \"00367384-f2ca-4817-a170-5720fcb0b1e3\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " Nov 03 22:40:32.696760 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696609 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-credential-socket\") pod \"00367384-f2ca-4817-a170-5720fcb0b1e3\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " Nov 03 22:40:32.696760 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696658 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-certs\") pod \"00367384-f2ca-4817-a170-5720fcb0b1e3\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " Nov 03 22:40:32.696760 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696709 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00367384-f2ca-4817-a170-5720fcb0b1e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00367384-f2ca-4817-a170-5720fcb0b1e3-istiod-ca-cert\") pod \"00367384-f2ca-4817-a170-5720fcb0b1e3\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " Nov 03 22:40:32.696919 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696767 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-kube-api-access-rlkzr\" (UniqueName: \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-kube-api-access-rlkzr\") pod \"00367384-f2ca-4817-a170-5720fcb0b1e3\" (UID: \"00367384-f2ca-4817-a170-5720fcb0b1e3\") " Nov 03 22:40:32.696970 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696927 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-credential-socket" pod "00367384-f2ca-4817-a170-5720fcb0b1e3" (UID: "00367384-f2ca-4817-a170-5720fcb0b1e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:32.696970 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.696948 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-certs" pod "00367384-f2ca-4817-a170-5720fcb0b1e3" (UID: "00367384-f2ca-4817-a170-5720fcb0b1e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:32.697178 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.697155 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00367384-f2ca-4817-a170-5720fcb0b1e3-istiod-ca-cert" pod "00367384-f2ca-4817-a170-5720fcb0b1e3" (UID: "00367384-f2ca-4817-a170-5720fcb0b1e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:32.697474 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.697450 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:32.697600 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.697480 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:32.697600 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.697521 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:32.697600 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.697538 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:32.697600 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.697553 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00367384-f2ca-4817-a170-5720fcb0b1e3-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:32.698809 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.698783 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-token" pod "00367384-f2ca-4817-a170-5720fcb0b1e3" (UID: "00367384-f2ca-4817-a170-5720fcb0b1e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:32.698923 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.698814 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-envoy" pod "00367384-f2ca-4817-a170-5720fcb0b1e3" (UID: "00367384-f2ca-4817-a170-5720fcb0b1e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:32.698967 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.698935 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-podinfo" pod "00367384-f2ca-4817-a170-5720fcb0b1e3" (UID: "00367384-f2ca-4817-a170-5720fcb0b1e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:40:32.699114 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.699096 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-kube-api-access-rlkzr" pod "00367384-f2ca-4817-a170-5720fcb0b1e3" (UID: "00367384-f2ca-4817-a170-5720fcb0b1e3"). InnerVolumeSpecName "kube-api-access-rlkzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:32.799545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.799487 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlkzr\" (UniqueName: \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-kube-api-access-rlkzr\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:32.799545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.799540 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:32.799733 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.799554 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:32.799733 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.799566 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00367384-f2ca-4817-a170-5720fcb0b1e3-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:32.981007 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.980924 6234 generic.go:356] "Generic (PLEG): container finished" podID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerID="d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982" exitCode=0 Nov 03 22:40:32.981007 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.980983 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" Nov 03 22:40:32.981007 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.980992 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" event={"ID":"00367384-f2ca-4817-a170-5720fcb0b1e3","Type":"ContainerDied","Data":"d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982"} Nov 03 22:40:32.981244 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.981049 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t" event={"ID":"00367384-f2ca-4817-a170-5720fcb0b1e3","Type":"ContainerDied","Data":"193be087b3b7e2d0ba36d1ed9aa22645b56d4f853e030457dce2592910ddd693"} Nov 03 22:40:32.981244 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.981073 6234 scope.go:122] "RemoveContainer" containerID="d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982" Nov 03 22:40:32.991249 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.991225 6234 scope.go:122] "RemoveContainer" containerID="d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982" Nov 03 22:40:32.991543 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:40:32.991522 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982\": container with ID starting with d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982 not found: ID does not exist" containerID="d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982" Nov 03 22:40:32.991613 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:32.991552 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982"} err="failed to get container status \"d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982\": rpc error: code = NotFound desc = could not find container \"d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982\": container with ID starting with d9a83a4bcb3c39915c5ae273c3d50fa36d840783114d64c0db4946fb27f1f982 not found: ID does not exist" Nov 03 22:40:33.135324 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:33.135287 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t"] Nov 03 22:40:33.138988 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:33.138961 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bzp9t"] Nov 03 22:40:34.706028 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:40:34.705997 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" path="/var/lib/kubelet/pods/00367384-f2ca-4817-a170-5720fcb0b1e3/volumes" Nov 03 22:40:58.997785 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:40:58.997743 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:41:28.997426 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:41:28.997343 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:41:42.611842 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:42.611799 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:41:51.612697 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:51.612654 6234 patch_prober.go:29] interesting pod/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:41:51.613153 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:51.612737 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.691375 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.691315 6234 generic.go:356] "Generic (PLEG): container finished" podID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerID="19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea" exitCode=0 Nov 03 22:41:55.692065 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.691538 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" event={"ID":"c6ab3852-4c26-42b9-9927-926bb9a00e79","Type":"ContainerDied","Data":"19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea"} Nov 03 22:41:55.692594 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.692571 6234 scope.go:122] "RemoveContainer" containerID="19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea" Nov 03 22:41:55.694597 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.694561 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:41:55.694700 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.694673 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:41:55.694765 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.694724 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:41:55.702151 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.702124 6234 generic.go:356] "Generic (PLEG): container finished" podID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerID="bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d" exitCode=0 Nov 03 22:41:55.702275 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.702186 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" event={"ID":"e7d62724-a2ac-43f0-8f35-9fc114109d79","Type":"ContainerDied","Data":"bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d"} Nov 03 22:41:55.702641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.702619 6234 scope.go:122] "RemoveContainer" containerID="bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d" Nov 03 22:41:55.709582 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.709337 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:41:55.710713 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.710002 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:41:55.710713 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:55.710075 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:41:56.713179 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:56.713105 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" event={"ID":"e7d62724-a2ac-43f0-8f35-9fc114109d79","Type":"ContainerStarted","Data":"2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1"} Nov 03 22:41:56.716401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:56.716368 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" event={"ID":"c6ab3852-4c26-42b9-9927-926bb9a00e79","Type":"ContainerStarted","Data":"a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db"} Nov 03 22:41:57.165580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:57.164166 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:41:57.178519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:57.169625 6234 patch_prober.go:29] interesting pod/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:41:57.178519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:57.169705 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 22:41:57.642624 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:57.642586 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:41:57.716927 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:57.716893 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:41:57.717839 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:57.717818 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:41:57.717959 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:57.717871 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 22:41:58.170457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:58.169970 6234 patch_prober.go:29] interesting pod/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:41:58.170457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:58.170041 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 22:41:58.717479 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:58.717444 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:41:58.717896 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:58.717584 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 22:41:59.015594 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:41:59.015482 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:41:59.163518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:59.162860 6234 patch_prober.go:29] interesting pod/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:41:59.163518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:59.162921 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 22:41:59.717333 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:59.717294 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:41:59.717566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:41:59.717348 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 22:42:00.188520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:00.182910 6234 patch_prober.go:29] interesting pod/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:42:00.188520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:00.182980 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 22:42:00.717452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:00.717422 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:42:00.717650 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:00.717469 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 22:42:01.188123 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:01.188001 6234 patch_prober.go:29] interesting pod/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:42:01.188527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:01.188383 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 22:42:01.716562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:01.716534 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:42:01.716854 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:01.716833 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 22:42:02.167650 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:02.167560 6234 patch_prober.go:29] interesting pod/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:42:02.168207 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:02.168173 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 22:42:02.721985 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:02.721954 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:42:02.722848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:02.722824 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:42:03.166462 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:03.166386 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:42:03.166912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:03.166897 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:42:03.167456 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:03.167435 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:42:12.195370 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:12.195335 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv"] Nov 03 22:42:12.195949 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:12.195646 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" containerID="cri-o://96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557" gracePeriod=30 Nov 03 22:42:12.413912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:12.413878 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv"] Nov 03 22:42:12.414214 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:12.414172 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" containerID="cri-o://af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0" gracePeriod=30 Nov 03 22:42:14.601565 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:14.601519 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:42:14.602024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:14.601596 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:42:17.551425 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.551400 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:42:17.659717 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.659368 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-credential-socket\") pod \"a41042da-424e-4520-8234-b11bd45a754d\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " Nov 03 22:42:17.659717 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.659426 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-kube-api-access-fjdkj\" (UniqueName: \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-kube-api-access-fjdkj\") pod \"a41042da-424e-4520-8234-b11bd45a754d\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " Nov 03 22:42:17.659717 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.659451 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a41042da-424e-4520-8234-b11bd45a754d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41042da-424e-4520-8234-b11bd45a754d-istio-podinfo\") pod \"a41042da-424e-4520-8234-b11bd45a754d\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " Nov 03 22:42:17.661810 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.660081 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-credential-socket" pod "a41042da-424e-4520-8234-b11bd45a754d" (UID: "a41042da-424e-4520-8234-b11bd45a754d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.666677 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.663588 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-certs\") pod \"a41042da-424e-4520-8234-b11bd45a754d\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " Nov 03 22:42:17.666677 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.663646 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41042da-424e-4520-8234-b11bd45a754d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41042da-424e-4520-8234-b11bd45a754d-istiod-ca-cert\") pod \"a41042da-424e-4520-8234-b11bd45a754d\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " Nov 03 22:42:17.666677 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.663710 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-istio-token\" (UniqueName: \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-istio-token\") pod \"a41042da-424e-4520-8234-b11bd45a754d\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " Nov 03 22:42:17.666677 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.666351 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41042da-424e-4520-8234-b11bd45a754d-istiod-ca-cert" pod "a41042da-424e-4520-8234-b11bd45a754d" (UID: "a41042da-424e-4520-8234-b11bd45a754d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:17.666949 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.666681 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-certs" pod "a41042da-424e-4520-8234-b11bd45a754d" (UID: "a41042da-424e-4520-8234-b11bd45a754d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.667539 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.667049 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-data\") pod \"a41042da-424e-4520-8234-b11bd45a754d\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " Nov 03 22:42:17.667539 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.667124 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-socket\") pod \"a41042da-424e-4520-8234-b11bd45a754d\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " Nov 03 22:42:17.667539 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.667158 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-envoy\") pod \"a41042da-424e-4520-8234-b11bd45a754d\" (UID: \"a41042da-424e-4520-8234-b11bd45a754d\") " Nov 03 22:42:17.671423 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.671180 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-data" pod "a41042da-424e-4520-8234-b11bd45a754d" (UID: "a41042da-424e-4520-8234-b11bd45a754d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.671423 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.671245 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-socket" pod "a41042da-424e-4520-8234-b11bd45a754d" (UID: "a41042da-424e-4520-8234-b11bd45a754d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.672750 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.672635 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.672987 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.672885 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.672987 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.672926 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.672987 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.672944 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.672987 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.672958 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41042da-424e-4520-8234-b11bd45a754d-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.674797 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.674760 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-kube-api-access-fjdkj" pod "a41042da-424e-4520-8234-b11bd45a754d" (UID: "a41042da-424e-4520-8234-b11bd45a754d"). InnerVolumeSpecName "kube-api-access-fjdkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:17.690557 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.689059 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-envoy" pod "a41042da-424e-4520-8234-b11bd45a754d" (UID: "a41042da-424e-4520-8234-b11bd45a754d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.727922 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.727856 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a41042da-424e-4520-8234-b11bd45a754d-istio-podinfo" pod "a41042da-424e-4520-8234-b11bd45a754d" (UID: "a41042da-424e-4520-8234-b11bd45a754d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:17.729738 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.728220 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-istio-token" pod "a41042da-424e-4520-8234-b11bd45a754d" (UID: "a41042da-424e-4520-8234-b11bd45a754d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:17.766197 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.766155 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:42:17.774471 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.774404 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjdkj\" (UniqueName: \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-kube-api-access-fjdkj\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.774471 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.774435 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41042da-424e-4520-8234-b11bd45a754d-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.774471 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.774451 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41042da-424e-4520-8234-b11bd45a754d-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.774471 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.774463 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41042da-424e-4520-8234-b11bd45a754d-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.875760 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.875721 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-credential-socket\") pod \"11f2da67-217f-4b4a-b606-deb7afc23093\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " Nov 03 22:42:17.875966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.875778 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-data\") pod \"11f2da67-217f-4b4a-b606-deb7afc23093\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " Nov 03 22:42:17.875966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.875849 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-envoy\") pod \"11f2da67-217f-4b4a-b606-deb7afc23093\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " Nov 03 22:42:17.875966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.875927 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-istio-token\" (UniqueName: \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-istio-token\") pod \"11f2da67-217f-4b4a-b606-deb7afc23093\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " Nov 03 22:42:17.876124 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.876001 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f2da67-217f-4b4a-b606-deb7afc23093-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f2da67-217f-4b4a-b606-deb7afc23093-istiod-ca-cert\") pod \"11f2da67-217f-4b4a-b606-deb7afc23093\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " Nov 03 22:42:17.876124 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.876043 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-certs\") pod \"11f2da67-217f-4b4a-b606-deb7afc23093\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " Nov 03 22:42:17.876124 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.876083 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-socket\") pod \"11f2da67-217f-4b4a-b606-deb7afc23093\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " Nov 03 22:42:17.876124 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.876023 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-credential-socket" pod "11f2da67-217f-4b4a-b606-deb7afc23093" (UID: "11f2da67-217f-4b4a-b606-deb7afc23093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.876124 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.876082 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-data" pod "11f2da67-217f-4b4a-b606-deb7afc23093" (UID: "11f2da67-217f-4b4a-b606-deb7afc23093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.876374 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.876317 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-kube-api-access-m44sd\" (UniqueName: \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-kube-api-access-m44sd\") pod \"11f2da67-217f-4b4a-b606-deb7afc23093\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " Nov 03 22:42:17.876374 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.876341 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-certs" pod "11f2da67-217f-4b4a-b606-deb7afc23093" (UID: "11f2da67-217f-4b4a-b606-deb7afc23093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.876374 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.876351 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-socket" pod "11f2da67-217f-4b4a-b606-deb7afc23093" (UID: "11f2da67-217f-4b4a-b606-deb7afc23093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.876535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.876394 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f2da67-217f-4b4a-b606-deb7afc23093-istiod-ca-cert" pod "11f2da67-217f-4b4a-b606-deb7afc23093" (UID: "11f2da67-217f-4b4a-b606-deb7afc23093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:17.876535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.876403 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11f2da67-217f-4b4a-b606-deb7afc23093-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f2da67-217f-4b4a-b606-deb7afc23093-istio-podinfo\") pod \"11f2da67-217f-4b4a-b606-deb7afc23093\" (UID: \"11f2da67-217f-4b4a-b606-deb7afc23093\") " Nov 03 22:42:17.877704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.877491 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f2da67-217f-4b4a-b606-deb7afc23093-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.877704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.877535 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.877704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.877549 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.877704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.877563 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.877704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.877578 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.878382 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.878355 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-istio-token" pod "11f2da67-217f-4b4a-b606-deb7afc23093" (UID: "11f2da67-217f-4b4a-b606-deb7afc23093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:17.878607 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.878580 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-kube-api-access-m44sd" pod "11f2da67-217f-4b4a-b606-deb7afc23093" (UID: "11f2da67-217f-4b4a-b606-deb7afc23093"). InnerVolumeSpecName "kube-api-access-m44sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:17.878607 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.878585 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-envoy" pod "11f2da67-217f-4b4a-b606-deb7afc23093" (UID: "11f2da67-217f-4b4a-b606-deb7afc23093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:17.878751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.878647 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11f2da67-217f-4b4a-b606-deb7afc23093-istio-podinfo" pod "11f2da67-217f-4b4a-b606-deb7afc23093" (UID: "11f2da67-217f-4b4a-b606-deb7afc23093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:17.949127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.949090 6234 generic.go:356] "Generic (PLEG): container finished" podID="11f2da67-217f-4b4a-b606-deb7afc23093" containerID="af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0" exitCode=0 Nov 03 22:42:17.949298 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.949158 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" Nov 03 22:42:17.949298 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.949174 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" event={"ID":"11f2da67-217f-4b4a-b606-deb7afc23093","Type":"ContainerDied","Data":"af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0"} Nov 03 22:42:17.949298 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.949234 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv" event={"ID":"11f2da67-217f-4b4a-b606-deb7afc23093","Type":"ContainerDied","Data":"cefefb3cfabe4498477745badf53eedff2590f84238d02b1cc81ff7495c9fc14"} Nov 03 22:42:17.949298 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.949265 6234 scope.go:122] "RemoveContainer" containerID="af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0" Nov 03 22:42:17.951190 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.951165 6234 generic.go:356] "Generic (PLEG): container finished" podID="a41042da-424e-4520-8234-b11bd45a754d" containerID="96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557" exitCode=0 Nov 03 22:42:17.951356 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.951189 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" event={"ID":"a41042da-424e-4520-8234-b11bd45a754d","Type":"ContainerDied","Data":"96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557"} Nov 03 22:42:17.951356 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.951229 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" Nov 03 22:42:17.951356 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.951243 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv" event={"ID":"a41042da-424e-4520-8234-b11bd45a754d","Type":"ContainerDied","Data":"bd4867a5aae22a93acc77126f76a3af7e91a2946a428199da05651aa054f390c"} Nov 03 22:42:17.960444 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.960424 6234 scope.go:122] "RemoveContainer" containerID="af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0" Nov 03 22:42:17.960799 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:42:17.960773 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0\": container with ID starting with af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0 not found: ID does not exist" containerID="af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0" Nov 03 22:42:17.960919 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.960808 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0"} err="failed to get container status \"af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0\": rpc error: code = NotFound desc = could not find container \"af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0\": container with ID starting with af5eaa09cd9560608035cc1a67d31528588ec6a03185b7113d2c450df85b2ae0 not found: ID does not exist" Nov 03 22:42:17.960919 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.960832 6234 scope.go:122] "RemoveContainer" containerID="96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557" Nov 03 22:42:17.972739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.972721 6234 scope.go:122] "RemoveContainer" containerID="96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557" Nov 03 22:42:17.973010 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:42:17.972994 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557\": container with ID starting with 96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557 not found: ID does not exist" containerID="96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557" Nov 03 22:42:17.973072 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.973017 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557"} err="failed to get container status \"96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557\": rpc error: code = NotFound desc = could not find container \"96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557\": container with ID starting with 96a116edc7d2350f4513a2efdca79d5ee3bb491dae565ca4cdae464fe5073557 not found: ID does not exist" Nov 03 22:42:17.978953 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.978927 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f2da67-217f-4b4a-b606-deb7afc23093-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.979049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.978956 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f2da67-217f-4b4a-b606-deb7afc23093-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.979049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.978971 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.979049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:17.978985 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m44sd\" (UniqueName: \"kubernetes.io/projected/11f2da67-217f-4b4a-b606-deb7afc23093-kube-api-access-m44sd\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.113003 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:18.112900 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv"] Nov 03 22:42:18.121922 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:18.121877 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-mkczv"] Nov 03 22:42:18.224688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:18.224659 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv"] Nov 03 22:42:18.229522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:18.229481 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-7tkvv"] Nov 03 22:42:18.705832 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:18.705798 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" path="/var/lib/kubelet/pods/11f2da67-217f-4b4a-b606-deb7afc23093/volumes" Nov 03 22:42:18.706318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:18.706301 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41042da-424e-4520-8234-b11bd45a754d" path="/var/lib/kubelet/pods/a41042da-424e-4520-8234-b11bd45a754d/volumes" Nov 03 22:42:22.014775 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.014735 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr"] Nov 03 22:42:22.016639 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.016610 6234 cpu_manager.go:414] "containerMap: removing container" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" Nov 03 22:42:22.016639 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.016639 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" Nov 03 22:42:22.016844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.016671 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" Nov 03 22:42:22.016844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.016686 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" Nov 03 22:42:22.016844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.016758 6234 cpu_manager.go:414] "containerMap: removing container" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" Nov 03 22:42:22.016844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.016768 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" Nov 03 22:42:22.017049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.017033 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f2da67-217f-4b4a-b606-deb7afc23093" containerName="istio-proxy" Nov 03 22:42:22.017102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.017066 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41042da-424e-4520-8234-b11bd45a754d" containerName="istio-proxy" Nov 03 22:42:22.017102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.017092 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="00367384-f2ca-4817-a170-5720fcb0b1e3" containerName="istio-proxy" Nov 03 22:42:22.022688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.022660 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.031047 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.030986 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.031153 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.031044 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df475\" (UniqueName: \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-kube-api-access-df475\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.031209 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.031154 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.031280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.031215 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbebf2e-04cf-4401-8562-29e70d767550-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.031280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.031259 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.031389 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.031287 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbebf2e-04cf-4401-8562-29e70d767550-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.031466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.031442 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.031541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.031488 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.031621 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.031602 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.118632 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.118592 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bc88eb47-openshift-default-dockercfg-h27hg\"" Nov 03 22:42:22.121947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.121917 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr"] Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.134788 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.134848 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df475\" (UniqueName: \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-kube-api-access-df475\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.134881 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.134911 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbebf2e-04cf-4401-8562-29e70d767550-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.134956 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.134992 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbebf2e-04cf-4401-8562-29e70d767550-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.135057 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.135105 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.135147 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.136366 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbebf2e-04cf-4401-8562-29e70d767550-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.137096 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.138522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.137669 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.139656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.138945 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.139656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.139169 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.142159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.142130 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.144468 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.144441 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbebf2e-04cf-4401-8562-29e70d767550-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.193208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.193172 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df475\" (UniqueName: \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-kube-api-access-df475\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.193208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.193206 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.336078 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.335983 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:22.804484 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.803830 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr"] Nov 03 22:42:22.847155 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:42:22.847120 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbebf2e_04cf_4401_8562_29e70d767550.slice/crio-58f19d514c2f77c6e3d7f89c259eba31a2c1babb8079945709976f390551e97d WatchSource:0}: Error finding container 58f19d514c2f77c6e3d7f89c259eba31a2c1babb8079945709976f390551e97d: Status 404 returned error can't find the container with id 58f19d514c2f77c6e3d7f89c259eba31a2c1babb8079945709976f390551e97d Nov 03 22:42:22.851289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.851250 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:22.851401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.851336 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:22.851401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.851382 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:22.992764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.992717 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" event={"ID":"6dbebf2e-04cf-4401-8562-29e70d767550","Type":"ContainerStarted","Data":"00dd24faf7ea9d14dfa1ea7a85a0371b61388e16c5e300f01f05c8ebdc817ce4"} Nov 03 22:42:22.992764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:22.992783 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" event={"ID":"6dbebf2e-04cf-4401-8562-29e70d767550","Type":"ContainerStarted","Data":"58f19d514c2f77c6e3d7f89c259eba31a2c1babb8079945709976f390551e97d"} Nov 03 22:42:23.108912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:23.108798 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" podStartSLOduration=2.108783299 podStartE2EDuration="2.108783299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:23.105516361 +0000 UTC m=+2062.517702372" watchObservedRunningTime="2026-11-03 22:42:23.108783299 +0000 UTC m=+2062.520969315" Nov 03 22:42:23.337153 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:23.337121 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:23.338472 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:23.338448 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:42:23.338562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:23.338520 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 22:42:24.337403 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:24.337361 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:42:24.337939 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:24.337423 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 22:42:25.336871 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.336837 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:42:25.337481 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.336891 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 22:42:25.868541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.865716 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg"] Nov 03 22:42:25.870467 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.870440 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:25.890601 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.890568 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr"] Nov 03 22:42:25.896568 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.896542 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:25.898648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.898618 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:25.898763 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.898668 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:25.898763 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.898710 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:25.898865 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.898799 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrp2d\" (UniqueName: \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-kube-api-access-hrp2d\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:25.898919 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.898874 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:25.899063 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.899046 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:25.899119 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.899099 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:25.899164 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.899142 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494f6493-e301-4f79-af6b-db4768de6a77-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:25.899210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.899192 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494f6493-e301-4f79-af6b-db4768de6a77-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:25.909092 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.909064 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg"] Nov 03 22:42:25.975400 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:25.975366 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr"] Nov 03 22:42:26.000424 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000389 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.000672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000446 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.000672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000533 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrp2d\" (UniqueName: \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-kube-api-access-hrp2d\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.000672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000557 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.000672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000627 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.000898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000685 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.000898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000740 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.000898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000775 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.000898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000848 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.001093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000910 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.001093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.000951 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.001093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.001008 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.001093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.001048 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494f6493-e301-4f79-af6b-db4768de6a77-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.001293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.001105 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494f6493-e301-4f79-af6b-db4768de6a77-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.001293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.001141 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wc7\" (UniqueName: \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-kube-api-access-s7wc7\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.001293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.001172 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.001293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.001196 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.001293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.001220 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.001743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.001714 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.002144 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.002105 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.002392 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.002369 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.002477 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.002458 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.003030 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.003003 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494f6493-e301-4f79-af6b-db4768de6a77-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.004850 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.004820 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.005345 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.005321 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494f6493-e301-4f79-af6b-db4768de6a77-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.078527 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.076114 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrp2d\" (UniqueName: \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-kube-api-access-hrp2d\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.090191 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.089014 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.102377 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.102323 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.102571 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.102463 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wc7\" (UniqueName: \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-kube-api-access-s7wc7\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.102571 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.102511 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.102910 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.102881 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.103011 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.102975 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.103542 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.103091 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.103542 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.103147 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.103542 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.103180 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.103542 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.103224 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.103542 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.103258 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.103542 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.103538 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.103928 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.103774 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.104101 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.104000 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.104101 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.104038 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.105817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.105791 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.105939 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.105924 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.166518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.166410 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.166853 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.166817 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wc7\" (UniqueName: \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-kube-api-access-s7wc7\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.211578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.199976 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:26.214762 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.213396 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:26.382892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.382854 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:42:26.383529 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.382917 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 22:42:26.830857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.830823 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg"] Nov 03 22:42:26.872001 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:42:26.871971 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb95e06_d3e2_4b7c_ad23_5bfbf2da70e1.slice/crio-89829ac5c5f11659b7709eebafb9817e35123cd9a4cf55c1444187101229a172 WatchSource:0}: Error finding container 89829ac5c5f11659b7709eebafb9817e35123cd9a4cf55c1444187101229a172: Status 404 returned error can't find the container with id 89829ac5c5f11659b7709eebafb9817e35123cd9a4cf55c1444187101229a172 Nov 03 22:42:26.875706 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.875670 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:26.875784 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.875759 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:26.875838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.875805 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:26.876632 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.876610 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr"] Nov 03 22:42:26.878554 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:42:26.878489 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494f6493_e301_4f79_af6b_db4768de6a77.slice/crio-337cf9724918b25041e83db9408f7c7fc2bfe35474bd1ef7dfa6592bf22d3c12 WatchSource:0}: Error finding container 337cf9724918b25041e83db9408f7c7fc2bfe35474bd1ef7dfa6592bf22d3c12: Status 404 returned error can't find the container with id 337cf9724918b25041e83db9408f7c7fc2bfe35474bd1ef7dfa6592bf22d3c12 Nov 03 22:42:26.881631 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.881602 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:26.881755 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.881675 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:26.881755 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:26.881722 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:27.076228 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.076185 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" event={"ID":"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1","Type":"ContainerStarted","Data":"f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362"} Nov 03 22:42:27.076404 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.076240 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" event={"ID":"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1","Type":"ContainerStarted","Data":"89829ac5c5f11659b7709eebafb9817e35123cd9a4cf55c1444187101229a172"} Nov 03 22:42:27.079988 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.079315 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" event={"ID":"494f6493-e301-4f79-af6b-db4768de6a77","Type":"ContainerStarted","Data":"eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4"} Nov 03 22:42:27.079988 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.079356 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" event={"ID":"494f6493-e301-4f79-af6b-db4768de6a77","Type":"ContainerStarted","Data":"337cf9724918b25041e83db9408f7c7fc2bfe35474bd1ef7dfa6592bf22d3c12"} Nov 03 22:42:27.188137 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.188047 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" podStartSLOduration=2.188029814 podStartE2EDuration="2.188029814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:27.186455255 +0000 UTC m=+2066.598641272" watchObservedRunningTime="2026-11-03 22:42:27.188029814 +0000 UTC m=+2066.600215832" Nov 03 22:42:27.200353 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.200316 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:27.201965 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.201938 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:42:27.202110 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.201995 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:42:27.215160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.215124 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:27.217645 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.217622 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:42:27.217763 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.217669 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:42:27.335285 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.335222 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" podStartSLOduration=2.33520103 podStartE2EDuration="2.33520103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:27.333847743 +0000 UTC m=+2066.746033750" watchObservedRunningTime="2026-11-03 22:42:27.33520103 +0000 UTC m=+2066.747387054" Nov 03 22:42:27.336713 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.336684 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:42:27.336829 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:27.336747 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 22:42:28.201024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:28.200983 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:42:28.201716 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:28.201677 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:42:28.215531 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:28.215106 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:42:28.215531 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:28.215160 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:42:28.338984 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:28.338948 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:42:28.339276 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:28.339231 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 22:42:29.075521 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:42:29.066710 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:42:29.206306 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:29.206270 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:42:29.206931 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:29.206901 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:42:29.217884 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:29.217780 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:42:29.217884 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:29.217838 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:42:29.337330 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:29.337253 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:42:29.337544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:29.337477 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 22:42:30.201097 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:30.201056 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:42:30.201291 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:30.201120 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:42:30.215059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:30.215030 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:42:30.215416 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:30.215075 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:42:30.621438 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:30.621348 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:30.622572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:30.622550 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:30.623922 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:30.623899 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:42:31.200442 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:31.200406 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:42:31.200670 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:31.200463 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:42:31.215745 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:31.215716 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:42:31.216110 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:31.215766 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:42:32.201035 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:32.201007 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:42:32.201247 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:32.201107 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:42:32.214874 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:32.214842 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:42:32.215024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:32.214896 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:42:33.201336 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:33.201301 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:42:33.201879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:33.201353 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 22:42:33.215568 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:33.215538 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 22:42:33.215699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:33.215591 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 22:42:34.233511 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:34.229192 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:34.233511 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:34.231667 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:34.233511 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:34.231784 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:42:34.301245 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:34.301211 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:34.301948 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:34.301922 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:34.303109 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:34.303077 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:42:37.149333 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.149289 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n"] Nov 03 22:42:37.154615 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.154589 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.196123 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.196097 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54"] Nov 03 22:42:37.199666 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.199645 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.222993 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.222965 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.223128 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223006 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.223128 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223034 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.223128 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223086 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.223311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223160 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.223311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223181 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.223311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223219 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7bb701-7f9b-4071-ba29-c319ec34818f-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.223311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223300 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.223520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223322 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.223520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223345 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.223520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223376 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.223520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223392 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.223520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223410 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvs8b\" (UniqueName: \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-kube-api-access-qvs8b\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.223520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223426 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.223520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223468 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppngv\" (UniqueName: \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-kube-api-access-ppngv\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.223892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223550 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.223892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223584 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.223892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.223601 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.244343 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.244315 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n"] Nov 03 22:42:37.259262 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.259233 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54"] Nov 03 22:42:37.324803 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.324772 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.324975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.324810 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.324975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.324834 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.324975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.324864 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.324975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.324883 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.324975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.324899 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvs8b\" (UniqueName: \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-kube-api-access-qvs8b\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.324975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.324914 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.324975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.324949 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppngv\" (UniqueName: \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-kube-api-access-ppngv\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.325372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325010 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.325372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325057 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.325372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325094 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.325372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325161 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.325372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325202 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.325372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325238 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.325372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325257 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.325372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325265 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.325372 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325314 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.325928 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325381 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.325928 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325410 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.325928 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325419 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.325928 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325470 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.325928 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325483 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7bb701-7f9b-4071-ba29-c319ec34818f-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.325928 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325709 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.326325 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.325939 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.326325 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.326055 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.326325 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.326179 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7bb701-7f9b-4071-ba29-c319ec34818f-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.326325 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.326263 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.326325 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.326316 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.327923 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.327902 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.328023 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.327901 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.328141 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.328122 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.328594 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.328573 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.380134 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.380072 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvs8b\" (UniqueName: \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-kube-api-access-qvs8b\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.380279 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.380250 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.380941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.380915 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppngv\" (UniqueName: \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-kube-api-access-ppngv\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.381061 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.381039 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.469382 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.469355 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:37.514333 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.514301 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:37.900232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.900144 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n"] Nov 03 22:42:37.904139 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:42:37.904112 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19a7d0a_f98a_46cf_919c_a247c438d5a5.slice/crio-aeacd886712cd1616cccc1d048e2dcac57e2fb43e00c29defc4870b25b24c33f WatchSource:0}: Error finding container aeacd886712cd1616cccc1d048e2dcac57e2fb43e00c29defc4870b25b24c33f: Status 404 returned error can't find the container with id aeacd886712cd1616cccc1d048e2dcac57e2fb43e00c29defc4870b25b24c33f Nov 03 22:42:37.907093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.907056 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:37.907195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.907136 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:37.907195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.907180 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:37.952988 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.952950 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54"] Nov 03 22:42:37.967882 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:42:37.967854 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7bb701_7f9b_4071_ba29_c319ec34818f.slice/crio-5759c1c0ec23b1eb4efa472044c5f29e789c706e99397f2428c5762f517dbd9d WatchSource:0}: Error finding container 5759c1c0ec23b1eb4efa472044c5f29e789c706e99397f2428c5762f517dbd9d: Status 404 returned error can't find the container with id 5759c1c0ec23b1eb4efa472044c5f29e789c706e99397f2428c5762f517dbd9d Nov 03 22:42:37.971245 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.971209 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:37.971341 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.971297 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:37.971386 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:37.971344 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:38.270568 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.270529 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" event={"ID":"4f7bb701-7f9b-4071-ba29-c319ec34818f","Type":"ContainerStarted","Data":"4fc74ffbc78233de6a9af63cf1ccad525417ac4311f8d2b57054dc84571b057c"} Nov 03 22:42:38.271123 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.270599 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" event={"ID":"4f7bb701-7f9b-4071-ba29-c319ec34818f","Type":"ContainerStarted","Data":"5759c1c0ec23b1eb4efa472044c5f29e789c706e99397f2428c5762f517dbd9d"} Nov 03 22:42:38.272941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.272908 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" event={"ID":"f19a7d0a-f98a-46cf-919c-a247c438d5a5","Type":"ContainerStarted","Data":"0d4d9ec73ef988a62b932f82d171837aa4e138efe236d731351b92dc00e2c94e"} Nov 03 22:42:38.273080 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.272964 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" event={"ID":"f19a7d0a-f98a-46cf-919c-a247c438d5a5","Type":"ContainerStarted","Data":"aeacd886712cd1616cccc1d048e2dcac57e2fb43e00c29defc4870b25b24c33f"} Nov 03 22:42:38.359516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.359071 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:42:38.382984 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.382929 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" podStartSLOduration=1.382914892 podStartE2EDuration="1.382914892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:38.381590548 +0000 UTC m=+2077.793776565" watchObservedRunningTime="2026-11-03 22:42:38.382914892 +0000 UTC m=+2077.795100908" Nov 03 22:42:38.469580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.469479 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:38.471010 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.470987 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:42:38.471106 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.471032 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:42:38.496028 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.495949 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" podStartSLOduration=1.49592524 podStartE2EDuration="1.49592524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:38.490986672 +0000 UTC m=+2077.903172701" watchObservedRunningTime="2026-11-03 22:42:38.49592524 +0000 UTC m=+2077.908111342" Nov 03 22:42:38.515669 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:38.515627 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:39.473523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:39.470040 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:42:39.473523 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:39.470104 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:42:39.516104 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:39.516065 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:42:39.516263 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:39.516123 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 22:42:40.505054 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:40.504573 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:42:40.505054 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:40.504641 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:42:40.526339 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:40.526297 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:42:40.526615 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:40.526583 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 22:42:41.488603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:41.475101 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:42:41.488603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:41.475165 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:42:41.515121 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:41.515083 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:42:41.515531 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:41.515136 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 22:42:42.472868 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:42.472831 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:42:42.473220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:42.473186 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:42:42.515417 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:42.515387 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:42:42.515804 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:42.515432 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 22:42:43.476911 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:43.476870 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:42:43.478419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:43.478384 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 22:42:43.514814 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:43.514783 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:42:43.514986 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:43.514830 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 22:42:44.515757 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:44.515721 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:42:44.516270 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:44.515786 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 22:42:44.711707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:44.711677 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:44.712197 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:44.712181 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:44.713193 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:44.713167 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:42:45.519735 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:45.519706 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:45.520211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:45.520163 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:45.520955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:45.520937 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:42:53.082426 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:53.082384 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp"] Nov 03 22:42:58.083082 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.083014 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" podUID="391cd6a8-2698-4fc8-9ede-b4290d03c0b0" containerName="payload-processing" containerID="cri-o://1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192" gracePeriod=25 Nov 03 22:42:58.357229 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.357204 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:42:58.486026 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.485990 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-kube-api-access-t6992\" (UniqueName: \"kubernetes.io/projected/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-kube-api-access-t6992\") pod \"391cd6a8-2698-4fc8-9ede-b4290d03c0b0\" (UID: \"391cd6a8-2698-4fc8-9ede-b4290d03c0b0\") " Nov 03 22:42:58.486331 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.486204 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-plugins-config-volume\") pod \"391cd6a8-2698-4fc8-9ede-b4290d03c0b0\" (UID: \"391cd6a8-2698-4fc8-9ede-b4290d03c0b0\") " Nov 03 22:42:58.486679 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.486649 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-plugins-config-volume" pod "391cd6a8-2698-4fc8-9ede-b4290d03c0b0" (UID: "391cd6a8-2698-4fc8-9ede-b4290d03c0b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:58.486941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.486923 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:58.488701 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.488668 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-kube-api-access-t6992" pod "391cd6a8-2698-4fc8-9ede-b4290d03c0b0" (UID: "391cd6a8-2698-4fc8-9ede-b4290d03c0b0"). InnerVolumeSpecName "kube-api-access-t6992". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:58.588414 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.588335 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6992\" (UniqueName: \"kubernetes.io/projected/391cd6a8-2698-4fc8-9ede-b4290d03c0b0-kube-api-access-t6992\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:58.612511 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.612466 6234 generic.go:356] "Generic (PLEG): container finished" podID="391cd6a8-2698-4fc8-9ede-b4290d03c0b0" containerID="1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192" exitCode=0 Nov 03 22:42:58.612695 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.612560 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" Nov 03 22:42:58.612695 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.612557 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" event={"ID":"391cd6a8-2698-4fc8-9ede-b4290d03c0b0","Type":"ContainerDied","Data":"1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192"} Nov 03 22:42:58.612695 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.612691 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp" event={"ID":"391cd6a8-2698-4fc8-9ede-b4290d03c0b0","Type":"ContainerDied","Data":"32bacd553ce966836c0af5a0f31a0f12f4457f529896bc6780d3ab647eb0ca8e"} Nov 03 22:42:58.612843 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.612721 6234 scope.go:122] "RemoveContainer" containerID="1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192" Nov 03 22:42:58.624810 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.624792 6234 scope.go:122] "RemoveContainer" containerID="1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192" Nov 03 22:42:58.625121 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:42:58.625104 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192\": container with ID starting with 1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192 not found: ID does not exist" containerID="1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192" Nov 03 22:42:58.625188 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.625135 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192"} err="failed to get container status \"1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192\": rpc error: code = NotFound desc = could not find container \"1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192\": container with ID starting with 1be5b9bbd3518bcbbc10d93f8421c667fc7dcc5a08ce93e14999270800233192 not found: ID does not exist" Nov 03 22:42:58.769407 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.769374 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp"] Nov 03 22:42:58.776907 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:42:58.776860 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-zjnjp"] Nov 03 22:42:59.002571 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:42:59.002536 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:43:00.728520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:43:00.725849 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391cd6a8-2698-4fc8-9ede-b4290d03c0b0" path="/var/lib/kubelet/pods/391cd6a8-2698-4fc8-9ede-b4290d03c0b0/volumes" Nov 03 22:43:25.492921 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:43:25.492885 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:43:28.998056 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:43:28.998018 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:43:59.019984 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:43:59.019942 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:44:25.054545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:25.054452 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n"] Nov 03 22:44:25.055127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:25.054682 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" containerID="cri-o://0d4d9ec73ef988a62b932f82d171837aa4e138efe236d731351b92dc00e2c94e" gracePeriod=30 Nov 03 22:44:25.067809 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:25.067780 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54"] Nov 03 22:44:25.068003 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:25.067982 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerName="istio-proxy" containerID="cri-o://4fc74ffbc78233de6a9af63cf1ccad525417ac4311f8d2b57054dc84571b057c" gracePeriod=30 Nov 03 22:44:25.096884 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:25.096842 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr"] Nov 03 22:44:25.097089 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:25.097053 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" containerID="cri-o://00dd24faf7ea9d14dfa1ea7a85a0371b61388e16c5e300f01f05c8ebdc817ce4" gracePeriod=30 Nov 03 22:44:28.997104 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:44:28.997066 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:44:29.714215 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:29.714182 6234 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:44:29.714422 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:29.714243 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:44:30.353187 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.353115 6234 generic.go:356] "Generic (PLEG): container finished" podID="6dbebf2e-04cf-4401-8562-29e70d767550" containerID="00dd24faf7ea9d14dfa1ea7a85a0371b61388e16c5e300f01f05c8ebdc817ce4" exitCode=0 Nov 03 22:44:30.353586 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.353170 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" event={"ID":"6dbebf2e-04cf-4401-8562-29e70d767550","Type":"ContainerDied","Data":"00dd24faf7ea9d14dfa1ea7a85a0371b61388e16c5e300f01f05c8ebdc817ce4"} Nov 03 22:44:30.355587 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.355559 6234 generic.go:356] "Generic (PLEG): container finished" podID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerID="0d4d9ec73ef988a62b932f82d171837aa4e138efe236d731351b92dc00e2c94e" exitCode=0 Nov 03 22:44:30.355740 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.355723 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" event={"ID":"f19a7d0a-f98a-46cf-919c-a247c438d5a5","Type":"ContainerDied","Data":"0d4d9ec73ef988a62b932f82d171837aa4e138efe236d731351b92dc00e2c94e"} Nov 03 22:44:30.357200 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.357171 6234 generic.go:356] "Generic (PLEG): container finished" podID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerID="4fc74ffbc78233de6a9af63cf1ccad525417ac4311f8d2b57054dc84571b057c" exitCode=0 Nov 03 22:44:30.357325 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.357211 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" event={"ID":"4f7bb701-7f9b-4071-ba29-c319ec34818f","Type":"ContainerDied","Data":"4fc74ffbc78233de6a9af63cf1ccad525417ac4311f8d2b57054dc84571b057c"} Nov 03 22:44:30.395488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.395414 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:44:30.429124 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.429102 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:44:30.443565 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.443541 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:44:30.497281 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497251 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-data\") pod \"6dbebf2e-04cf-4401-8562-29e70d767550\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " Nov 03 22:44:30.497642 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497312 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-data\") pod \"4f7bb701-7f9b-4071-ba29-c319ec34818f\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " Nov 03 22:44:30.497642 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497344 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-istio-token\" (UniqueName: \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-istio-token\") pod \"6dbebf2e-04cf-4401-8562-29e70d767550\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " Nov 03 22:44:30.497642 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497395 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-kube-api-access-ppngv\" (UniqueName: \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-kube-api-access-ppngv\") pod \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " Nov 03 22:44:30.497642 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497424 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-credential-socket\") pod \"6dbebf2e-04cf-4401-8562-29e70d767550\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " Nov 03 22:44:30.497642 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497450 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-token\" (UniqueName: \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-token\") pod \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " Nov 03 22:44:30.497642 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497551 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-envoy\") pod \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " Nov 03 22:44:30.497642 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497607 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbebf2e-04cf-4401-8562-29e70d767550-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbebf2e-04cf-4401-8562-29e70d767550-istiod-ca-cert\") pod \"6dbebf2e-04cf-4401-8562-29e70d767550\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " Nov 03 22:44:30.498070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497672 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-kube-api-access-qvs8b\" (UniqueName: \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-kube-api-access-qvs8b\") pod \"4f7bb701-7f9b-4071-ba29-c319ec34818f\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " Nov 03 22:44:30.498070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497721 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-credential-socket\") pod \"4f7bb701-7f9b-4071-ba29-c319ec34818f\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " Nov 03 22:44:30.498070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497771 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-kube-api-access-df475\" (UniqueName: \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-kube-api-access-df475\") pod \"6dbebf2e-04cf-4401-8562-29e70d767550\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " Nov 03 22:44:30.498070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497801 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-certs\") pod \"6dbebf2e-04cf-4401-8562-29e70d767550\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " Nov 03 22:44:30.498070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497855 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-certs\") pod \"4f7bb701-7f9b-4071-ba29-c319ec34818f\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " Nov 03 22:44:30.498070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497983 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istiod-ca-cert\") pod \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " Nov 03 22:44:30.498070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498025 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-socket\") pod \"4f7bb701-7f9b-4071-ba29-c319ec34818f\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " Nov 03 22:44:30.498070 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498055 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-data\") pod \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498083 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-envoy\") pod \"6dbebf2e-04cf-4401-8562-29e70d767550\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498113 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-certs\") pod \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498141 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-podinfo\") pod \"4f7bb701-7f9b-4071-ba29-c319ec34818f\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498192 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-credential-socket\") pod \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498229 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dbebf2e-04cf-4401-8562-29e70d767550-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbebf2e-04cf-4401-8562-29e70d767550-istio-podinfo\") pod \"6dbebf2e-04cf-4401-8562-29e70d767550\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498258 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-podinfo\") pod \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498317 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-socket\") pod \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\" (UID: \"f19a7d0a-f98a-46cf-919c-a247c438d5a5\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498345 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7bb701-7f9b-4071-ba29-c319ec34818f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7bb701-7f9b-4071-ba29-c319ec34818f-istiod-ca-cert\") pod \"4f7bb701-7f9b-4071-ba29-c319ec34818f\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498411 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-token\" (UniqueName: \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-token\") pod \"4f7bb701-7f9b-4071-ba29-c319ec34818f\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498456 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-socket\") pod \"6dbebf2e-04cf-4401-8562-29e70d767550\" (UID: \"6dbebf2e-04cf-4401-8562-29e70d767550\") " Nov 03 22:44:30.498513 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498511 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-envoy\") pod \"4f7bb701-7f9b-4071-ba29-c319ec34818f\" (UID: \"4f7bb701-7f9b-4071-ba29-c319ec34818f\") " Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497667 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-data" pod "4f7bb701-7f9b-4071-ba29-c319ec34818f" (UID: "4f7bb701-7f9b-4071-ba29-c319ec34818f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497683 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-data" pod "6dbebf2e-04cf-4401-8562-29e70d767550" (UID: "6dbebf2e-04cf-4401-8562-29e70d767550"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.497855 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-credential-socket" pod "6dbebf2e-04cf-4401-8562-29e70d767550" (UID: "6dbebf2e-04cf-4401-8562-29e70d767550"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498079 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-credential-socket" pod "4f7bb701-7f9b-4071-ba29-c319ec34818f" (UID: "4f7bb701-7f9b-4071-ba29-c319ec34818f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498011 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbebf2e-04cf-4401-8562-29e70d767550-istiod-ca-cert" pod "6dbebf2e-04cf-4401-8562-29e70d767550" (UID: "6dbebf2e-04cf-4401-8562-29e70d767550"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498286 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-certs" pod "6dbebf2e-04cf-4401-8562-29e70d767550" (UID: "6dbebf2e-04cf-4401-8562-29e70d767550"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498371 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istiod-ca-cert" pod "f19a7d0a-f98a-46cf-919c-a247c438d5a5" (UID: "f19a7d0a-f98a-46cf-919c-a247c438d5a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498631 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-certs" pod "4f7bb701-7f9b-4071-ba29-c319ec34818f" (UID: "4f7bb701-7f9b-4071-ba29-c319ec34818f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498824 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-credential-socket" pod "f19a7d0a-f98a-46cf-919c-a247c438d5a5" (UID: "f19a7d0a-f98a-46cf-919c-a247c438d5a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.498979 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-socket" pod "4f7bb701-7f9b-4071-ba29-c319ec34818f" (UID: "4f7bb701-7f9b-4071-ba29-c319ec34818f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.499140 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-data" pod "f19a7d0a-f98a-46cf-919c-a247c438d5a5" (UID: "f19a7d0a-f98a-46cf-919c-a247c438d5a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.499974 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.499687 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-certs" pod "f19a7d0a-f98a-46cf-919c-a247c438d5a5" (UID: "f19a7d0a-f98a-46cf-919c-a247c438d5a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.500689 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.499991 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-socket" pod "f19a7d0a-f98a-46cf-919c-a247c438d5a5" (UID: "f19a7d0a-f98a-46cf-919c-a247c438d5a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.500689 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.500206 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-socket" pod "6dbebf2e-04cf-4401-8562-29e70d767550" (UID: "6dbebf2e-04cf-4401-8562-29e70d767550"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.500689 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.500568 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-istio-token" pod "6dbebf2e-04cf-4401-8562-29e70d767550" (UID: "6dbebf2e-04cf-4401-8562-29e70d767550"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.500689 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.500624 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-kube-api-access-qvs8b" pod "4f7bb701-7f9b-4071-ba29-c319ec34818f" (UID: "4f7bb701-7f9b-4071-ba29-c319ec34818f"). InnerVolumeSpecName "kube-api-access-qvs8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.500910 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.500733 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-envoy" pod "f19a7d0a-f98a-46cf-919c-a247c438d5a5" (UID: "f19a7d0a-f98a-46cf-919c-a247c438d5a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.501534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.501192 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-kube-api-access-ppngv" pod "f19a7d0a-f98a-46cf-919c-a247c438d5a5" (UID: "f19a7d0a-f98a-46cf-919c-a247c438d5a5"). InnerVolumeSpecName "kube-api-access-ppngv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.501534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.501210 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7bb701-7f9b-4071-ba29-c319ec34818f-istiod-ca-cert" pod "4f7bb701-7f9b-4071-ba29-c319ec34818f" (UID: "4f7bb701-7f9b-4071-ba29-c319ec34818f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.501534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.501593 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-token" pod "f19a7d0a-f98a-46cf-919c-a247c438d5a5" (UID: "f19a7d0a-f98a-46cf-919c-a247c438d5a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.502445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.502407 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-kube-api-access-df475" pod "6dbebf2e-04cf-4401-8562-29e70d767550" (UID: "6dbebf2e-04cf-4401-8562-29e70d767550"). InnerVolumeSpecName "kube-api-access-df475". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.502445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.502435 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-envoy" pod "4f7bb701-7f9b-4071-ba29-c319ec34818f" (UID: "4f7bb701-7f9b-4071-ba29-c319ec34818f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.502617 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.502530 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-podinfo" pod "4f7bb701-7f9b-4071-ba29-c319ec34818f" (UID: "4f7bb701-7f9b-4071-ba29-c319ec34818f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:30.502692 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.502625 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-podinfo" pod "f19a7d0a-f98a-46cf-919c-a247c438d5a5" (UID: "f19a7d0a-f98a-46cf-919c-a247c438d5a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:30.502844 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.502822 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-token" pod "4f7bb701-7f9b-4071-ba29-c319ec34818f" (UID: "4f7bb701-7f9b-4071-ba29-c319ec34818f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.503012 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.502996 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-envoy" pod "6dbebf2e-04cf-4401-8562-29e70d767550" (UID: "6dbebf2e-04cf-4401-8562-29e70d767550"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.503290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.503269 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dbebf2e-04cf-4401-8562-29e70d767550-istio-podinfo" pod "6dbebf2e-04cf-4401-8562-29e70d767550" (UID: "6dbebf2e-04cf-4401-8562-29e70d767550"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:30.601102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601070 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601097 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601109 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601120 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601128 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601137 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601146 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601154 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbebf2e-04cf-4401-8562-29e70d767550-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601162 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601170 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601178 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7bb701-7f9b-4071-ba29-c319ec34818f-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601186 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601194 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601205 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601214 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601222 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601232 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601239 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppngv\" (UniqueName: \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-kube-api-access-ppngv\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601249 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601258 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601266 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19a7d0a-f98a-46cf-919c-a247c438d5a5-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601274 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbebf2e-04cf-4401-8562-29e70d767550-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601283 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvs8b\" (UniqueName: \"kubernetes.io/projected/4f7bb701-7f9b-4071-ba29-c319ec34818f-kube-api-access-qvs8b\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601291 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601300 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df475\" (UniqueName: \"kubernetes.io/projected/6dbebf2e-04cf-4401-8562-29e70d767550-kube-api-access-df475\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601307 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbebf2e-04cf-4401-8562-29e70d767550-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.601365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:30.601317 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7bb701-7f9b-4071-ba29-c319ec34818f-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:31.365483 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.365383 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" Nov 03 22:44:31.365983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.365385 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n" event={"ID":"f19a7d0a-f98a-46cf-919c-a247c438d5a5","Type":"ContainerDied","Data":"aeacd886712cd1616cccc1d048e2dcac57e2fb43e00c29defc4870b25b24c33f"} Nov 03 22:44:31.365983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.365600 6234 scope.go:122] "RemoveContainer" containerID="0d4d9ec73ef988a62b932f82d171837aa4e138efe236d731351b92dc00e2c94e" Nov 03 22:44:31.367461 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.367221 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" Nov 03 22:44:31.367811 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.367786 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54" event={"ID":"4f7bb701-7f9b-4071-ba29-c319ec34818f","Type":"ContainerDied","Data":"5759c1c0ec23b1eb4efa472044c5f29e789c706e99397f2428c5762f517dbd9d"} Nov 03 22:44:31.369739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.369713 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" event={"ID":"6dbebf2e-04cf-4401-8562-29e70d767550","Type":"ContainerDied","Data":"58f19d514c2f77c6e3d7f89c259eba31a2c1babb8079945709976f390551e97d"} Nov 03 22:44:31.369872 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.369826 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr" Nov 03 22:44:31.382076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.382051 6234 scope.go:122] "RemoveContainer" containerID="4fc74ffbc78233de6a9af63cf1ccad525417ac4311f8d2b57054dc84571b057c" Nov 03 22:44:31.392350 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.392332 6234 scope.go:122] "RemoveContainer" containerID="00dd24faf7ea9d14dfa1ea7a85a0371b61388e16c5e300f01f05c8ebdc817ce4" Nov 03 22:44:31.568834 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.568800 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr"] Nov 03 22:44:31.614881 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.614847 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-8sxbr"] Nov 03 22:44:31.676583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.676457 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54"] Nov 03 22:44:31.680535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.680506 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-j2n54"] Nov 03 22:44:31.783150 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.783021 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n"] Nov 03 22:44:31.787072 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:31.787044 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-mmb8n"] Nov 03 22:44:32.706876 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:32.706843 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" path="/var/lib/kubelet/pods/4f7bb701-7f9b-4071-ba29-c319ec34818f/volumes" Nov 03 22:44:32.707380 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:32.707366 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" path="/var/lib/kubelet/pods/6dbebf2e-04cf-4401-8562-29e70d767550/volumes" Nov 03 22:44:32.707842 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:32.707829 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" path="/var/lib/kubelet/pods/f19a7d0a-f98a-46cf-919c-a247c438d5a5/volumes" Nov 03 22:44:40.281909 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:40.281870 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x"] Nov 03 22:44:40.296057 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:40.296025 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx"] Nov 03 22:44:47.767525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:47.767478 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b"] Nov 03 22:44:47.814235 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:47.814204 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm"] Nov 03 22:44:48.833000 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:48.832971 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm"] Nov 03 22:44:52.717667 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:52.717632 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:44:53.833825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:53.833773 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" podUID="d1819764-af2f-4666-8b13-bc4bda3a1f09" containerName="payload-pre-processing" containerID="cri-o://3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781" gracePeriod=25 Nov 03 22:44:54.181693 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.181273 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:44:54.299975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.298669 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1819764-af2f-4666-8b13-bc4bda3a1f09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1819764-af2f-4666-8b13-bc4bda3a1f09-plugins-config-volume\") pod \"d1819764-af2f-4666-8b13-bc4bda3a1f09\" (UID: \"d1819764-af2f-4666-8b13-bc4bda3a1f09\") " Nov 03 22:44:54.299975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.298977 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1819764-af2f-4666-8b13-bc4bda3a1f09-kube-api-access-8qlpn\" (UniqueName: \"kubernetes.io/projected/d1819764-af2f-4666-8b13-bc4bda3a1f09-kube-api-access-8qlpn\") pod \"d1819764-af2f-4666-8b13-bc4bda3a1f09\" (UID: \"d1819764-af2f-4666-8b13-bc4bda3a1f09\") " Nov 03 22:44:54.301102 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.301069 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1819764-af2f-4666-8b13-bc4bda3a1f09-plugins-config-volume" pod "d1819764-af2f-4666-8b13-bc4bda3a1f09" (UID: "d1819764-af2f-4666-8b13-bc4bda3a1f09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:54.302924 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.302864 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1819764-af2f-4666-8b13-bc4bda3a1f09-kube-api-access-8qlpn" pod "d1819764-af2f-4666-8b13-bc4bda3a1f09" (UID: "d1819764-af2f-4666-8b13-bc4bda3a1f09"). InnerVolumeSpecName "kube-api-access-8qlpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:54.369183 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.369140 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g"] Nov 03 22:44:54.403065 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.402971 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qlpn\" (UniqueName: \"kubernetes.io/projected/d1819764-af2f-4666-8b13-bc4bda3a1f09-kube-api-access-8qlpn\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:54.403314 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.403288 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1819764-af2f-4666-8b13-bc4bda3a1f09-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:54.470970 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.470915 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss"] Nov 03 22:44:54.535667 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.535639 6234 generic.go:356] "Generic (PLEG): container finished" podID="d1819764-af2f-4666-8b13-bc4bda3a1f09" containerID="3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781" exitCode=0 Nov 03 22:44:54.535849 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.535834 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" event={"ID":"d1819764-af2f-4666-8b13-bc4bda3a1f09","Type":"ContainerDied","Data":"3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781"} Nov 03 22:44:54.535938 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.535927 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" event={"ID":"d1819764-af2f-4666-8b13-bc4bda3a1f09","Type":"ContainerDied","Data":"4e169869e5c99ee5ddc7bd27f08295bc443594aca2f5002d850a99799993e497"} Nov 03 22:44:54.536022 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.536014 6234 scope.go:122] "RemoveContainer" containerID="3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781" Nov 03 22:44:54.536182 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.536172 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm" Nov 03 22:44:54.557610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.557586 6234 scope.go:122] "RemoveContainer" containerID="3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781" Nov 03 22:44:54.557970 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:44:54.557943 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781\": container with ID starting with 3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781 not found: ID does not exist" containerID="3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781" Nov 03 22:44:54.558043 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.557986 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781"} err="failed to get container status \"3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781\": rpc error: code = NotFound desc = could not find container \"3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781\": container with ID starting with 3f4ef9cf519cd5652ef39bd37fa51e59384c8a498d5b22dfcdb309d03f6b4781 not found: ID does not exist" Nov 03 22:44:54.731390 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.731360 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm"] Nov 03 22:44:54.734929 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:54.734900 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6a6b7f-c7d4dd94c-krcxm"] Nov 03 22:44:55.310520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:55.308729 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" containerID="cri-o://8c22107d623634afe4ec674382669f6a9212c15454272ce640999ba1a55218d1" gracePeriod=45 Nov 03 22:44:55.319682 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:55.319644 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" containerID="cri-o://7be4254ab1ebfe2502ec4830b2fa1ca30287546a27fdc258d4df55e9547df3eb" gracePeriod=45 Nov 03 22:44:55.378515 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:55.366316 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused" Nov 03 22:44:55.509521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:55.501623 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": dial tcp 10.134.0.62:8000: connect: connection refused" Nov 03 22:44:55.621574 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:55.620887 6234 generic.go:356] "Generic (PLEG): container finished" podID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerID="7be4254ab1ebfe2502ec4830b2fa1ca30287546a27fdc258d4df55e9547df3eb" exitCode=0 Nov 03 22:44:55.621574 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:55.620987 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" event={"ID":"7df4efee-bc6e-4a65-b97b-5228c8a9d408","Type":"ContainerDied","Data":"7be4254ab1ebfe2502ec4830b2fa1ca30287546a27fdc258d4df55e9547df3eb"} Nov 03 22:44:55.633338 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:55.633041 6234 generic.go:356] "Generic (PLEG): container finished" podID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerID="8c22107d623634afe4ec674382669f6a9212c15454272ce640999ba1a55218d1" exitCode=0 Nov 03 22:44:55.633338 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:55.633135 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" event={"ID":"99b65a57-f9eb-40e0-989d-7655c3cc3d6e","Type":"ContainerDied","Data":"8c22107d623634afe4ec674382669f6a9212c15454272ce640999ba1a55218d1"} Nov 03 22:44:56.063669 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.053670 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4vztp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:56.063669 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.053732 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:56.512484 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.512448 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:44:56.545721 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.545691 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:44:56.661903 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.661811 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" event={"ID":"7df4efee-bc6e-4a65-b97b-5228c8a9d408","Type":"ContainerDied","Data":"33aa05f9d909d87fe1e681bb48d7e2813a5d522f0a1c4d0e94295c77b7c93955"} Nov 03 22:44:56.680653 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.680629 6234 scope.go:122] "RemoveContainer" containerID="7be4254ab1ebfe2502ec4830b2fa1ca30287546a27fdc258d4df55e9547df3eb" Nov 03 22:44:56.680965 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.662155 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" Nov 03 22:44:56.685342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.685315 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" event={"ID":"99b65a57-f9eb-40e0-989d-7655c3cc3d6e","Type":"ContainerDied","Data":"da43171faa15d0b874925ca0d14016560d4b7e6db2c5a60812de2d63f2c9a32c"} Nov 03 22:44:56.685787 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.685768 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x" Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.723908 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-dshm\") pod \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.723986 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-kube-api-access-mkhtr\" (UniqueName: \"kubernetes.io/projected/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-kube-api-access-mkhtr\") pod \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.724036 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-home\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-home\") pod \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.724076 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-model-cache\") pod \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.724136 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-home\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-home\") pod \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.724166 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tls-certs\" (UniqueName: \"kubernetes.io/secret/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tls-certs\") pod \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.724277 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df4efee-bc6e-4a65-b97b-5228c8a9d408-kube-api-access-2jrcc\" (UniqueName: \"kubernetes.io/projected/7df4efee-bc6e-4a65-b97b-5228c8a9d408-kube-api-access-2jrcc\") pod \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.724317 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tls-certs\") pod \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.724375 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tmp-dir\") pod \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.724480 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-dshm\") pod \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\" (UID: \"7df4efee-bc6e-4a65-b97b-5228c8a9d408\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.724627 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tmp-dir\") pod \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " Nov 03 22:44:56.725257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.724664 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-model-cache\") pod \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\" (UID: \"99b65a57-f9eb-40e0-989d-7655c3cc3d6e\") " Nov 03 22:44:56.725992 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.725571 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-model-cache" pod "99b65a57-f9eb-40e0-989d-7655c3cc3d6e" (UID: "99b65a57-f9eb-40e0-989d-7655c3cc3d6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:56.727594 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.727570 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-home" pod "7df4efee-bc6e-4a65-b97b-5228c8a9d408" (UID: "7df4efee-bc6e-4a65-b97b-5228c8a9d408"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:56.727947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.727767 6234 scope.go:122] "RemoveContainer" containerID="8c22107d623634afe4ec674382669f6a9212c15454272ce640999ba1a55218d1" Nov 03 22:44:56.731097 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.728039 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-model-cache" pod "7df4efee-bc6e-4a65-b97b-5228c8a9d408" (UID: "7df4efee-bc6e-4a65-b97b-5228c8a9d408"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:56.731206 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.728881 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-dshm" pod "99b65a57-f9eb-40e0-989d-7655c3cc3d6e" (UID: "99b65a57-f9eb-40e0-989d-7655c3cc3d6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:56.731206 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.730222 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tmp-dir" pod "7df4efee-bc6e-4a65-b97b-5228c8a9d408" (UID: "7df4efee-bc6e-4a65-b97b-5228c8a9d408"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:56.731206 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.730631 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-home" pod "99b65a57-f9eb-40e0-989d-7655c3cc3d6e" (UID: "99b65a57-f9eb-40e0-989d-7655c3cc3d6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:56.731206 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.730771 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-kube-api-access-mkhtr" pod "99b65a57-f9eb-40e0-989d-7655c3cc3d6e" (UID: "99b65a57-f9eb-40e0-989d-7655c3cc3d6e"). InnerVolumeSpecName "kube-api-access-mkhtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:56.731206 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.731004 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tmp-dir" pod "99b65a57-f9eb-40e0-989d-7655c3cc3d6e" (UID: "99b65a57-f9eb-40e0-989d-7655c3cc3d6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:56.731206 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.729999 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1819764-af2f-4666-8b13-bc4bda3a1f09" path="/var/lib/kubelet/pods/d1819764-af2f-4666-8b13-bc4bda3a1f09/volumes" Nov 03 22:44:56.731526 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.731308 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df4efee-bc6e-4a65-b97b-5228c8a9d408-kube-api-access-2jrcc" pod "7df4efee-bc6e-4a65-b97b-5228c8a9d408" (UID: "7df4efee-bc6e-4a65-b97b-5228c8a9d408"). InnerVolumeSpecName "kube-api-access-2jrcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:56.734466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.734432 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tls-certs" pod "99b65a57-f9eb-40e0-989d-7655c3cc3d6e" (UID: "99b65a57-f9eb-40e0-989d-7655c3cc3d6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:56.735397 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.735375 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-dshm" pod "7df4efee-bc6e-4a65-b97b-5228c8a9d408" (UID: "7df4efee-bc6e-4a65-b97b-5228c8a9d408"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:56.736012 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.735967 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tls-certs" pod "7df4efee-bc6e-4a65-b97b-5228c8a9d408" (UID: "7df4efee-bc6e-4a65-b97b-5228c8a9d408"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:56.828895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.828857 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.828895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.828895 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.829127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.828909 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.829127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.828923 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.829127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.828937 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhtr\" (UniqueName: \"kubernetes.io/projected/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-kube-api-access-mkhtr\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.829127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.828950 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.829127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.828963 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.829127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.828975 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.829127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.828989 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.829127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.829000 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jrcc\" (UniqueName: \"kubernetes.io/projected/7df4efee-bc6e-4a65-b97b-5228c8a9d408-kube-api-access-2jrcc\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.829127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.829014 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b65a57-f9eb-40e0-989d-7655c3cc3d6e-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:56.829127 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:56.829028 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df4efee-bc6e-4a65-b97b-5228c8a9d408-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:57.097525 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:57.097479 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx"] Nov 03 22:44:57.103166 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:57.103138 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx"] Nov 03 22:44:57.207815 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:57.207781 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x"] Nov 03 22:44:57.221373 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:57.221340 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6a6b7f/distinct-w6-6a6b7f-kserve-7cd788788c-k9p9x"] Nov 03 22:44:57.377793 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:57.377703 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6a6b7f/premium-w6-6a6b7f-kserve-74c9b6858d-xr8nx" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.705095 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:58.705054 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" path="/var/lib/kubelet/pods/7df4efee-bc6e-4a65-b97b-5228c8a9d408/volumes" Nov 03 22:44:58.705709 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:58.705690 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" path="/var/lib/kubelet/pods/99b65a57-f9eb-40e0-989d-7655c3cc3d6e/volumes" Nov 03 22:44:58.996779 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:44:58.996688 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:44:59.370441 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:59.370350 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" podUID="5dc0ea6a-242f-44a2-aa04-e6b2c3b22735" containerName="payload-processing" containerID="cri-o://bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d" gracePeriod=25 Nov 03 22:44:59.472292 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:44:59.472250 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" podUID="3ea59b41-386c-489e-811c-01f107d9458f" containerName="payload-pre-processing" containerID="cri-o://6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db" gracePeriod=25 Nov 03 22:45:00.185457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.185418 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj"] Nov 03 22:45:00.186149 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186128 6234 cpu_manager.go:414] "containerMap: removing container" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" Nov 03 22:45:00.186149 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186150 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186177 6234 cpu_manager.go:414] "containerMap: removing container" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerName="istio-proxy" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186183 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerName="istio-proxy" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186203 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186209 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186226 6234 cpu_manager.go:414] "containerMap: removing container" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186231 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186243 6234 cpu_manager.go:414] "containerMap: removing container" podUID="391cd6a8-2698-4fc8-9ede-b4290d03c0b0" containerName="payload-processing" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186248 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391cd6a8-2698-4fc8-9ede-b4290d03c0b0" containerName="payload-processing" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186255 6234 cpu_manager.go:414] "containerMap: removing container" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186259 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186277 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d1819764-af2f-4666-8b13-bc4bda3a1f09" containerName="payload-pre-processing" Nov 03 22:45:00.186294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186282 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1819764-af2f-4666-8b13-bc4bda3a1f09" containerName="payload-pre-processing" Nov 03 22:45:00.186958 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186370 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b65a57-f9eb-40e0-989d-7655c3cc3d6e" containerName="main" Nov 03 22:45:00.186958 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186381 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7bb701-7f9b-4071-ba29-c319ec34818f" containerName="istio-proxy" Nov 03 22:45:00.186958 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186388 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbebf2e-04cf-4401-8562-29e70d767550" containerName="istio-proxy" Nov 03 22:45:00.186958 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186395 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19a7d0a-f98a-46cf-919c-a247c438d5a5" containerName="istio-proxy" Nov 03 22:45:00.186958 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186401 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="391cd6a8-2698-4fc8-9ede-b4290d03c0b0" containerName="payload-processing" Nov 03 22:45:00.186958 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186406 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df4efee-bc6e-4a65-b97b-5228c8a9d408" containerName="main" Nov 03 22:45:00.186958 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.186413 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1819764-af2f-4666-8b13-bc4bda3a1f09" containerName="payload-pre-processing" Nov 03 22:45:00.192050 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.192030 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" Nov 03 22:45:00.202381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.202330 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q"] Nov 03 22:45:00.207262 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.207242 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr"] Nov 03 22:45:00.207371 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.207354 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" Nov 03 22:45:00.211358 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.211339 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" Nov 03 22:45:00.274417 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.274380 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d85139-0754-4092-8367-75e4dc27ee8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj\" (UID: \"48d85139-0754-4092-8367-75e4dc27ee8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" Nov 03 22:45:00.274625 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.274433 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlxdr\" (UniqueName: \"kubernetes.io/projected/48d85139-0754-4092-8367-75e4dc27ee8f-kube-api-access-rlxdr\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj\" (UID: \"48d85139-0754-4092-8367-75e4dc27ee8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" Nov 03 22:45:00.325605 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.325563 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:45:00.352975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.352937 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr"] Nov 03 22:45:00.373396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.373354 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj"] Nov 03 22:45:00.376739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.376707 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d85139-0754-4092-8367-75e4dc27ee8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj\" (UID: \"48d85139-0754-4092-8367-75e4dc27ee8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" Nov 03 22:45:00.376902 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.376762 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6sx\" (UniqueName: \"kubernetes.io/projected/2791a167-71b5-4ee0-b76d-1e28f47cf03f-kube-api-access-vl6sx\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr\" (UID: \"2791a167-71b5-4ee0-b76d-1e28f47cf03f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" Nov 03 22:45:00.376902 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.376804 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlxdr\" (UniqueName: \"kubernetes.io/projected/48d85139-0754-4092-8367-75e4dc27ee8f-kube-api-access-rlxdr\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj\" (UID: \"48d85139-0754-4092-8367-75e4dc27ee8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" Nov 03 22:45:00.377050 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.376992 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a392641-2369-47eb-9d75-e7316cf47589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q\" (UID: \"3a392641-2369-47eb-9d75-e7316cf47589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" Nov 03 22:45:00.377050 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.377034 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlh9b\" (UniqueName: \"kubernetes.io/projected/3a392641-2369-47eb-9d75-e7316cf47589-kube-api-access-xlh9b\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q\" (UID: \"3a392641-2369-47eb-9d75-e7316cf47589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" Nov 03 22:45:00.377269 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.377085 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2791a167-71b5-4ee0-b76d-1e28f47cf03f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr\" (UID: \"2791a167-71b5-4ee0-b76d-1e28f47cf03f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" Nov 03 22:45:00.378088 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.378062 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d85139-0754-4092-8367-75e4dc27ee8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj\" (UID: \"48d85139-0754-4092-8367-75e4dc27ee8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" Nov 03 22:45:00.435069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.435032 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlxdr\" (UniqueName: \"kubernetes.io/projected/48d85139-0754-4092-8367-75e4dc27ee8f-kube-api-access-rlxdr\") pod \"maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj\" (UID: \"48d85139-0754-4092-8367-75e4dc27ee8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" Nov 03 22:45:00.436636 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.436567 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q"] Nov 03 22:45:00.478552 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.478523 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a392641-2369-47eb-9d75-e7316cf47589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q\" (UID: \"3a392641-2369-47eb-9d75-e7316cf47589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" Nov 03 22:45:00.478699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.478563 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlh9b\" (UniqueName: \"kubernetes.io/projected/3a392641-2369-47eb-9d75-e7316cf47589-kube-api-access-xlh9b\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q\" (UID: \"3a392641-2369-47eb-9d75-e7316cf47589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" Nov 03 22:45:00.478699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.478605 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2791a167-71b5-4ee0-b76d-1e28f47cf03f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr\" (UID: \"2791a167-71b5-4ee0-b76d-1e28f47cf03f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" Nov 03 22:45:00.478831 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.478741 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6sx\" (UniqueName: \"kubernetes.io/projected/2791a167-71b5-4ee0-b76d-1e28f47cf03f-kube-api-access-vl6sx\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr\" (UID: \"2791a167-71b5-4ee0-b76d-1e28f47cf03f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" Nov 03 22:45:00.479232 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.479173 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2791a167-71b5-4ee0-b76d-1e28f47cf03f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr\" (UID: \"2791a167-71b5-4ee0-b76d-1e28f47cf03f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" Nov 03 22:45:00.479324 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.479234 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a392641-2369-47eb-9d75-e7316cf47589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q\" (UID: \"3a392641-2369-47eb-9d75-e7316cf47589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" Nov 03 22:45:00.508981 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.508960 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" Nov 03 22:45:00.534307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.534278 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6sx\" (UniqueName: \"kubernetes.io/projected/2791a167-71b5-4ee0-b76d-1e28f47cf03f-kube-api-access-vl6sx\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr\" (UID: \"2791a167-71b5-4ee0-b76d-1e28f47cf03f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" Nov 03 22:45:00.534688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.534659 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlh9b\" (UniqueName: \"kubernetes.io/projected/3a392641-2369-47eb-9d75-e7316cf47589-kube-api-access-xlh9b\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q\" (UID: \"3a392641-2369-47eb-9d75-e7316cf47589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" Nov 03 22:45:00.537272 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.537249 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" Nov 03 22:45:00.823741 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.823672 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" Nov 03 22:45:00.888526 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.887020 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj"] Nov 03 22:45:00.893650 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:45:00.893618 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d85139_0754_4092_8367_75e4dc27ee8f.slice/crio-ba501f5954d5f92513b502d4a1888d441562a60516da025db49f9e5a79ecd2f5 WatchSource:0}: Error finding container ba501f5954d5f92513b502d4a1888d441562a60516da025db49f9e5a79ecd2f5: Status 404 returned error can't find the container with id ba501f5954d5f92513b502d4a1888d441562a60516da025db49f9e5a79ecd2f5 Nov 03 22:45:00.896676 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.895813 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:45:00.941279 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:00.941168 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr"] Nov 03 22:45:00.944775 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:45:00.944747 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2791a167_71b5_4ee0_b76d_1e28f47cf03f.slice/crio-00e2b9005c37c557e44f43640f54adb8200d885fe3de99e46ebbdda6180dfefb WatchSource:0}: Error finding container 00e2b9005c37c557e44f43640f54adb8200d885fe3de99e46ebbdda6180dfefb: Status 404 returned error can't find the container with id 00e2b9005c37c557e44f43640f54adb8200d885fe3de99e46ebbdda6180dfefb Nov 03 22:45:01.195816 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:01.195779 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q"] Nov 03 22:45:01.200329 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:45:01.200293 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a392641_2369_47eb_9d75_e7316cf47589.slice/crio-576b0ee426d0eba603549332fb19f3435bf5921f6addccd929e0fb80bd50534a WatchSource:0}: Error finding container 576b0ee426d0eba603549332fb19f3435bf5921f6addccd929e0fb80bd50534a: Status 404 returned error can't find the container with id 576b0ee426d0eba603549332fb19f3435bf5921f6addccd929e0fb80bd50534a Nov 03 22:45:01.723471 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:01.723410 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" event={"ID":"2791a167-71b5-4ee0-b76d-1e28f47cf03f","Type":"ContainerStarted","Data":"00e2b9005c37c557e44f43640f54adb8200d885fe3de99e46ebbdda6180dfefb"} Nov 03 22:45:01.725053 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:01.725020 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" event={"ID":"48d85139-0754-4092-8367-75e4dc27ee8f","Type":"ContainerStarted","Data":"ba501f5954d5f92513b502d4a1888d441562a60516da025db49f9e5a79ecd2f5"} Nov 03 22:45:01.726683 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:01.726656 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" event={"ID":"3a392641-2369-47eb-9d75-e7316cf47589","Type":"ContainerStarted","Data":"576b0ee426d0eba603549332fb19f3435bf5921f6addccd929e0fb80bd50534a"} Nov 03 22:45:02.788862 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:02.788821 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" containerID="cri-o://547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0" gracePeriod=45 Nov 03 22:45:02.834996 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:02.834960 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" containerID="cri-o://bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184" gracePeriod=45 Nov 03 22:45:03.132961 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.132914 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:45:03.139404 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.139376 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:45:03.222271 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.222236 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-dshm\") pod \"c0d1460a-5288-4f03-8159-34390ce36678\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " Nov 03 22:45:03.222474 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.222285 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-model-cache\") pod \"c0d1460a-5288-4f03-8159-34390ce36678\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " Nov 03 22:45:03.222474 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.222413 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d1460a-5288-4f03-8159-34390ce36678-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d1460a-5288-4f03-8159-34390ce36678-tls-certs\") pod \"c0d1460a-5288-4f03-8159-34390ce36678\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " Nov 03 22:45:03.222474 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.222457 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-tmp-dir\") pod \"c0d1460a-5288-4f03-8159-34390ce36678\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " Nov 03 22:45:03.222682 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.222512 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d1460a-5288-4f03-8159-34390ce36678-kube-api-access-p5kxt\" (UniqueName: \"kubernetes.io/projected/c0d1460a-5288-4f03-8159-34390ce36678-kube-api-access-p5kxt\") pod \"c0d1460a-5288-4f03-8159-34390ce36678\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " Nov 03 22:45:03.222682 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.222559 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-home\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-home\") pod \"c0d1460a-5288-4f03-8159-34390ce36678\" (UID: \"c0d1460a-5288-4f03-8159-34390ce36678\") " Nov 03 22:45:03.222682 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.222598 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-model-cache" pod "c0d1460a-5288-4f03-8159-34390ce36678" (UID: "c0d1460a-5288-4f03-8159-34390ce36678"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.222871 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.222822 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-home" pod "c0d1460a-5288-4f03-8159-34390ce36678" (UID: "c0d1460a-5288-4f03-8159-34390ce36678"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.222871 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.222839 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-tmp-dir" pod "c0d1460a-5288-4f03-8159-34390ce36678" (UID: "c0d1460a-5288-4f03-8159-34390ce36678"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.223200 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.223176 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.223311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.223213 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.223311 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.223223 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.224661 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.224634 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-dshm" pod "c0d1460a-5288-4f03-8159-34390ce36678" (UID: "c0d1460a-5288-4f03-8159-34390ce36678"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.224782 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.224723 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d1460a-5288-4f03-8159-34390ce36678-tls-certs" pod "c0d1460a-5288-4f03-8159-34390ce36678" (UID: "c0d1460a-5288-4f03-8159-34390ce36678"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:03.224845 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.224827 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d1460a-5288-4f03-8159-34390ce36678-kube-api-access-p5kxt" pod "c0d1460a-5288-4f03-8159-34390ce36678" (UID: "c0d1460a-5288-4f03-8159-34390ce36678"). InnerVolumeSpecName "kube-api-access-p5kxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:03.324547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.324516 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-model-cache\") pod \"e33680db-536e-4c35-b6ac-1765204cc302\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " Nov 03 22:45:03.324709 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.324598 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e33680db-536e-4c35-b6ac-1765204cc302-tls-certs\" (UniqueName: \"kubernetes.io/secret/e33680db-536e-4c35-b6ac-1765204cc302-tls-certs\") pod \"e33680db-536e-4c35-b6ac-1765204cc302\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " Nov 03 22:45:03.324753 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.324715 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-dshm\") pod \"e33680db-536e-4c35-b6ac-1765204cc302\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " Nov 03 22:45:03.324785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.324777 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33680db-536e-4c35-b6ac-1765204cc302-kube-api-access-wrhbw\" (UniqueName: \"kubernetes.io/projected/e33680db-536e-4c35-b6ac-1765204cc302-kube-api-access-wrhbw\") pod \"e33680db-536e-4c35-b6ac-1765204cc302\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " Nov 03 22:45:03.324819 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.324784 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-model-cache" pod "e33680db-536e-4c35-b6ac-1765204cc302" (UID: "e33680db-536e-4c35-b6ac-1765204cc302"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.324854 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.324828 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-tmp-dir\") pod \"e33680db-536e-4c35-b6ac-1765204cc302\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " Nov 03 22:45:03.324895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.324872 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-home\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-home\") pod \"e33680db-536e-4c35-b6ac-1765204cc302\" (UID: \"e33680db-536e-4c35-b6ac-1765204cc302\") " Nov 03 22:45:03.325121 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.325095 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-tmp-dir" pod "e33680db-536e-4c35-b6ac-1765204cc302" (UID: "e33680db-536e-4c35-b6ac-1765204cc302"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.325282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.325189 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-home" pod "e33680db-536e-4c35-b6ac-1765204cc302" (UID: "e33680db-536e-4c35-b6ac-1765204cc302"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.325639 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.325617 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.325751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.325644 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.325751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.325659 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d1460a-5288-4f03-8159-34390ce36678-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.325751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.325675 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.325751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.325691 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d1460a-5288-4f03-8159-34390ce36678-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.325751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.325708 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5kxt\" (UniqueName: \"kubernetes.io/projected/c0d1460a-5288-4f03-8159-34390ce36678-kube-api-access-p5kxt\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.326879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.326860 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-dshm" pod "e33680db-536e-4c35-b6ac-1765204cc302" (UID: "e33680db-536e-4c35-b6ac-1765204cc302"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.326973 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.326911 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33680db-536e-4c35-b6ac-1765204cc302-tls-certs" pod "e33680db-536e-4c35-b6ac-1765204cc302" (UID: "e33680db-536e-4c35-b6ac-1765204cc302"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:03.327139 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.327124 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33680db-536e-4c35-b6ac-1765204cc302-kube-api-access-wrhbw" pod "e33680db-536e-4c35-b6ac-1765204cc302" (UID: "e33680db-536e-4c35-b6ac-1765204cc302"). InnerVolumeSpecName "kube-api-access-wrhbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:03.427563 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.427462 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33680db-536e-4c35-b6ac-1765204cc302-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.427563 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.427507 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33680db-536e-4c35-b6ac-1765204cc302-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.427563 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.427522 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrhbw\" (UniqueName: \"kubernetes.io/projected/e33680db-536e-4c35-b6ac-1765204cc302-kube-api-access-wrhbw\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.741203 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.741161 6234 generic.go:356] "Generic (PLEG): container finished" podID="c0d1460a-5288-4f03-8159-34390ce36678" containerID="547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0" exitCode=0 Nov 03 22:45:03.741381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.741243 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" Nov 03 22:45:03.741381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.741322 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" event={"ID":"c0d1460a-5288-4f03-8159-34390ce36678","Type":"ContainerDied","Data":"547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0"} Nov 03 22:45:03.741381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.741360 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" event={"ID":"c0d1460a-5288-4f03-8159-34390ce36678","Type":"ContainerDied","Data":"96170f51eddf35a74f43aacb1888569610c7944dadc6c22a3a7b144059664388"} Nov 03 22:45:03.741600 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.741388 6234 scope.go:122] "RemoveContainer" containerID="547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0" Nov 03 22:45:03.743812 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.743783 6234 generic.go:356] "Generic (PLEG): container finished" podID="e33680db-536e-4c35-b6ac-1765204cc302" containerID="bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184" exitCode=0 Nov 03 22:45:03.743983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.743942 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" event={"ID":"e33680db-536e-4c35-b6ac-1765204cc302","Type":"ContainerDied","Data":"bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184"} Nov 03 22:45:03.744051 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.743985 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" event={"ID":"e33680db-536e-4c35-b6ac-1765204cc302","Type":"ContainerDied","Data":"cf52841bff28752f6f319587614f3aacadac4bc31f2ac9b09351f16ef27ba61c"} Nov 03 22:45:03.744090 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.744050 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm" Nov 03 22:45:03.755282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.755261 6234 scope.go:122] "RemoveContainer" containerID="547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0" Nov 03 22:45:03.755636 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:03.755611 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0\": container with ID starting with 547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0 not found: ID does not exist" containerID="547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0" Nov 03 22:45:03.755744 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.755646 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0"} err="failed to get container status \"547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0\": rpc error: code = NotFound desc = could not find container \"547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0\": container with ID starting with 547e8e7299d19dc0b45255181b45d122f994d57c06101fbc371c96529744fbc0 not found: ID does not exist" Nov 03 22:45:03.755744 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.755671 6234 scope.go:122] "RemoveContainer" containerID="bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184" Nov 03 22:45:03.768060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.768039 6234 scope.go:122] "RemoveContainer" containerID="bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184" Nov 03 22:45:03.768366 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:03.768336 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184\": container with ID starting with bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184 not found: ID does not exist" containerID="bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184" Nov 03 22:45:03.768441 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.768377 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184"} err="failed to get container status \"bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184\": rpc error: code = NotFound desc = could not find container \"bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184\": container with ID starting with bba3904f67ee6125f726c3e1312e4e211e55e15bcf0a5dc2d029a54998995184 not found: ID does not exist" Nov 03 22:45:03.897465 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.897397 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm"] Nov 03 22:45:03.943837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:03.943769 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/simulator-w1-3378fd-kserve-55848f8dd9-6wcrm"] Nov 03 22:45:04.046606 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:04.046523 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b"] Nov 03 22:45:04.049848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:04.049815 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b"] Nov 03 22:45:04.070413 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:04.070370 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/premium-w1-3378fd-kserve-8474fb444-zdb8b" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 22:45:04.707957 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:04.707920 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d1460a-5288-4f03-8159-34390ce36678" path="/var/lib/kubelet/pods/c0d1460a-5288-4f03-8159-34390ce36678/volumes" Nov 03 22:45:04.708566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:04.708541 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e33680db-536e-4c35-b6ac-1765204cc302" path="/var/lib/kubelet/pods/e33680db-536e-4c35-b6ac-1765204cc302/volumes" Nov 03 22:45:05.763632 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:05.763606 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" event={"ID":"3a392641-2369-47eb-9d75-e7316cf47589","Type":"ContainerStarted","Data":"241a2fa0c24ffa254abd60501ed7486a1588594ea0e1be364093acb0016717bd"} Nov 03 22:45:05.765326 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:05.765296 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" event={"ID":"2791a167-71b5-4ee0-b76d-1e28f47cf03f","Type":"ContainerStarted","Data":"f3d2b68665803fc65f0ac8ecda65ad3ab9df1818c0bd2cfa108dc5af75d613d3"} Nov 03 22:45:05.870574 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:05.870522 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" podStartSLOduration=1.960463397 podStartE2EDuration="5.870507586s" totalImagesPullingTime="3.910044189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:05.869034439 +0000 UTC m=+2225.281220457" watchObservedRunningTime="2026-11-03 22:45:05.870507586 +0000 UTC m=+2225.282693621" Nov 03 22:45:05.974136 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:05.974072 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" podStartSLOduration=1.452423796 podStartE2EDuration="5.974053462s" totalImagesPullingTime="4.521629666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:05.973802686 +0000 UTC m=+2225.385988705" watchObservedRunningTime="2026-11-03 22:45:05.974053462 +0000 UTC m=+2225.386239480" Nov 03 22:45:06.773621 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:06.773579 6234 generic.go:356] "Generic (PLEG): container finished" podID="3a392641-2369-47eb-9d75-e7316cf47589" containerID="241a2fa0c24ffa254abd60501ed7486a1588594ea0e1be364093acb0016717bd" exitCode=60 Nov 03 22:45:06.774106 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:06.773854 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" event={"ID":"3a392641-2369-47eb-9d75-e7316cf47589","Type":"ContainerDied","Data":"241a2fa0c24ffa254abd60501ed7486a1588594ea0e1be364093acb0016717bd"} Nov 03 22:45:06.774506 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:06.774467 6234 scope.go:122] "RemoveContainer" containerID="241a2fa0c24ffa254abd60501ed7486a1588594ea0e1be364093acb0016717bd" Nov 03 22:45:06.785034 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:06.785000 6234 generic.go:356] "Generic (PLEG): container finished" podID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerID="f3d2b68665803fc65f0ac8ecda65ad3ab9df1818c0bd2cfa108dc5af75d613d3" exitCode=60 Nov 03 22:45:06.785157 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:06.785067 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" event={"ID":"2791a167-71b5-4ee0-b76d-1e28f47cf03f","Type":"ContainerDied","Data":"f3d2b68665803fc65f0ac8ecda65ad3ab9df1818c0bd2cfa108dc5af75d613d3"} Nov 03 22:45:06.785663 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:06.785642 6234 scope.go:122] "RemoveContainer" containerID="f3d2b68665803fc65f0ac8ecda65ad3ab9df1818c0bd2cfa108dc5af75d613d3" Nov 03 22:45:07.504836 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:07.504804 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" podUID="3ea59b41-386c-489e-811c-01f107d9458f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:45:07.505081 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:07.505059 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" podUID="5dc0ea6a-242f-44a2-aa04-e6b2c3b22735" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:45:07.794000 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:07.793968 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" event={"ID":"48d85139-0754-4092-8367-75e4dc27ee8f","Type":"ContainerStarted","Data":"50eb74016b28fd86862e4e6e173c5972fb6832b9c5a86df24474273ba941ce71"} Nov 03 22:45:07.796304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:07.796272 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" event={"ID":"3a392641-2369-47eb-9d75-e7316cf47589","Type":"ContainerStarted","Data":"70ad3ff6974ec1e2c5da87a7ee6435a82ce2ca6734f908eb3b588bd3347e7fea"} Nov 03 22:45:07.798243 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:07.798220 6234 generic.go:356] "Generic (PLEG): container finished" podID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerID="e9ef60efc6fa35ee65f8845664a787ec2030ef29210ab2a67ca5b6eae747daab" exitCode=60 Nov 03 22:45:07.798364 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:07.798272 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" event={"ID":"2791a167-71b5-4ee0-b76d-1e28f47cf03f","Type":"ContainerDied","Data":"e9ef60efc6fa35ee65f8845664a787ec2030ef29210ab2a67ca5b6eae747daab"} Nov 03 22:45:07.798364 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:07.798312 6234 scope.go:122] "RemoveContainer" containerID="f3d2b68665803fc65f0ac8ecda65ad3ab9df1818c0bd2cfa108dc5af75d613d3" Nov 03 22:45:07.798532 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:07.798514 6234 scope.go:122] "RemoveContainer" containerID="e9ef60efc6fa35ee65f8845664a787ec2030ef29210ab2a67ca5b6eae747daab" Nov 03 22:45:07.798836 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:07.798816 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr_odh-ai-gateway-infra(2791a167-71b5-4ee0-b76d-1e28f47cf03f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" Nov 03 22:45:07.901117 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:07.901025 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" podStartSLOduration=1.434024779 podStartE2EDuration="7.901011939s" totalImagesPullingTime="6.46698716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:07.899362751 +0000 UTC m=+2227.311548772" watchObservedRunningTime="2026-11-03 22:45:07.901011939 +0000 UTC m=+2227.313197955" Nov 03 22:45:08.807895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:08.807860 6234 scope.go:122] "RemoveContainer" containerID="e9ef60efc6fa35ee65f8845664a787ec2030ef29210ab2a67ca5b6eae747daab" Nov 03 22:45:08.808315 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:08.808239 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr_odh-ai-gateway-infra(2791a167-71b5-4ee0-b76d-1e28f47cf03f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" Nov 03 22:45:08.809434 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:08.809410 6234 generic.go:356] "Generic (PLEG): container finished" podID="48d85139-0754-4092-8367-75e4dc27ee8f" containerID="50eb74016b28fd86862e4e6e173c5972fb6832b9c5a86df24474273ba941ce71" exitCode=60 Nov 03 22:45:08.809558 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:08.809535 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" event={"ID":"48d85139-0754-4092-8367-75e4dc27ee8f","Type":"ContainerDied","Data":"50eb74016b28fd86862e4e6e173c5972fb6832b9c5a86df24474273ba941ce71"} Nov 03 22:45:08.809758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:08.809744 6234 scope.go:122] "RemoveContainer" containerID="50eb74016b28fd86862e4e6e173c5972fb6832b9c5a86df24474273ba941ce71" Nov 03 22:45:08.812732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:08.812714 6234 generic.go:356] "Generic (PLEG): container finished" podID="3a392641-2369-47eb-9d75-e7316cf47589" containerID="70ad3ff6974ec1e2c5da87a7ee6435a82ce2ca6734f908eb3b588bd3347e7fea" exitCode=60 Nov 03 22:45:08.812815 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:08.812765 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" event={"ID":"3a392641-2369-47eb-9d75-e7316cf47589","Type":"ContainerDied","Data":"70ad3ff6974ec1e2c5da87a7ee6435a82ce2ca6734f908eb3b588bd3347e7fea"} Nov 03 22:45:08.812815 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:08.812811 6234 scope.go:122] "RemoveContainer" containerID="241a2fa0c24ffa254abd60501ed7486a1588594ea0e1be364093acb0016717bd" Nov 03 22:45:08.813076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:08.813055 6234 scope.go:122] "RemoveContainer" containerID="70ad3ff6974ec1e2c5da87a7ee6435a82ce2ca6734f908eb3b588bd3347e7fea" Nov 03 22:45:08.813412 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:08.813382 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q_odh-ai-gateway-infra(3a392641-2369-47eb-9d75-e7316cf47589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" podUID="3a392641-2369-47eb-9d75-e7316cf47589" Nov 03 22:45:09.821749 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:09.821716 6234 generic.go:356] "Generic (PLEG): container finished" podID="48d85139-0754-4092-8367-75e4dc27ee8f" containerID="9a429924f3f8c49ea84091e2265c36ee11f96d33af90675882bace54ac3957ea" exitCode=60 Nov 03 22:45:09.822135 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:09.821798 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" event={"ID":"48d85139-0754-4092-8367-75e4dc27ee8f","Type":"ContainerDied","Data":"9a429924f3f8c49ea84091e2265c36ee11f96d33af90675882bace54ac3957ea"} Nov 03 22:45:09.822135 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:09.821854 6234 scope.go:122] "RemoveContainer" containerID="50eb74016b28fd86862e4e6e173c5972fb6832b9c5a86df24474273ba941ce71" Nov 03 22:45:09.822261 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:09.822182 6234 scope.go:122] "RemoveContainer" containerID="9a429924f3f8c49ea84091e2265c36ee11f96d33af90675882bace54ac3957ea" Nov 03 22:45:09.822571 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:09.822551 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj_odh-ai-gateway-infra(48d85139-0754-4092-8367-75e4dc27ee8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" Nov 03 22:45:09.823848 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:09.823830 6234 scope.go:122] "RemoveContainer" containerID="70ad3ff6974ec1e2c5da87a7ee6435a82ce2ca6734f908eb3b588bd3347e7fea" Nov 03 22:45:09.824167 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:09.824148 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q_odh-ai-gateway-infra(3a392641-2369-47eb-9d75-e7316cf47589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" podUID="3a392641-2369-47eb-9d75-e7316cf47589" Nov 03 22:45:10.831764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:10.831736 6234 scope.go:122] "RemoveContainer" containerID="9a429924f3f8c49ea84091e2265c36ee11f96d33af90675882bace54ac3957ea" Nov 03 22:45:10.832147 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:10.832036 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj_odh-ai-gateway-infra(48d85139-0754-4092-8367-75e4dc27ee8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" Nov 03 22:45:17.515800 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:17.515757 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" podUID="5dc0ea6a-242f-44a2-aa04-e6b2c3b22735" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:45:17.517955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:17.517922 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" podUID="3ea59b41-386c-489e-811c-01f107d9458f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:45:19.644850 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.644821 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv"] Nov 03 22:45:19.645519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.645488 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" Nov 03 22:45:19.645576 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.645522 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" Nov 03 22:45:19.645576 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.645571 6234 cpu_manager.go:414] "containerMap: removing container" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" Nov 03 22:45:19.645646 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.645578 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" Nov 03 22:45:19.645718 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.645708 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33680db-536e-4c35-b6ac-1765204cc302" containerName="main" Nov 03 22:45:19.645750 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.645727 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d1460a-5288-4f03-8159-34390ce36678" containerName="main" Nov 03 22:45:19.648221 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.647955 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:45:19.649049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.649032 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.697834 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.697804 6234 scope.go:122] "RemoveContainer" containerID="e9ef60efc6fa35ee65f8845664a787ec2030ef29210ab2a67ca5b6eae747daab" Nov 03 22:45:19.713541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.713514 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv"] Nov 03 22:45:19.782299 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.782266 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:45:19.785304 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.785279 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-plugins-config-volume\") pod \"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735\" (UID: \"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735\") " Nov 03 22:45:19.785424 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.785394 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-kube-api-access-rvh2t\" (UniqueName: \"kubernetes.io/projected/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-kube-api-access-rvh2t\") pod \"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735\" (UID: \"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735\") " Nov 03 22:45:19.785717 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.785698 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.785809 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.785723 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-plugins-config-volume" pod "5dc0ea6a-242f-44a2-aa04-e6b2c3b22735" (UID: "5dc0ea6a-242f-44a2-aa04-e6b2c3b22735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:19.785809 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.785743 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fa3225-0006-485d-a7de-59dd408b6f99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.785951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.785929 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.786013 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.785981 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.786086 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.786062 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.786219 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.786199 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.786308 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.786282 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l994b\" (UniqueName: \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-kube-api-access-l994b\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.786450 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.786434 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.786541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.786525 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.786643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.786629 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:19.787721 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.787703 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-kube-api-access-rvh2t" pod "5dc0ea6a-242f-44a2-aa04-e6b2c3b22735" (UID: "5dc0ea6a-242f-44a2-aa04-e6b2c3b22735"). InnerVolumeSpecName "kube-api-access-rvh2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:19.887548 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.887436 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea59b41-386c-489e-811c-01f107d9458f-kube-api-access-sj8gr\" (UniqueName: \"kubernetes.io/projected/3ea59b41-386c-489e-811c-01f107d9458f-kube-api-access-sj8gr\") pod \"3ea59b41-386c-489e-811c-01f107d9458f\" (UID: \"3ea59b41-386c-489e-811c-01f107d9458f\") " Nov 03 22:45:19.887695 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.887603 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea59b41-386c-489e-811c-01f107d9458f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea59b41-386c-489e-811c-01f107d9458f-plugins-config-volume\") pod \"3ea59b41-386c-489e-811c-01f107d9458f\" (UID: \"3ea59b41-386c-489e-811c-01f107d9458f\") " Nov 03 22:45:19.888005 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.887975 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.888132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888011 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea59b41-386c-489e-811c-01f107d9458f-plugins-config-volume" pod "3ea59b41-386c-489e-811c-01f107d9458f" (UID: "3ea59b41-386c-489e-811c-01f107d9458f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:19.888132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888030 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fa3225-0006-485d-a7de-59dd408b6f99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.888132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888113 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.888314 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888151 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.888314 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888188 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.888314 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888287 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.888445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888343 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l994b\" (UniqueName: \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-kube-api-access-l994b\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.888445 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888390 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.888531 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888441 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.888599 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888581 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea59b41-386c-489e-811c-01f107d9458f-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:19.888647 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.888605 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvh2t\" (UniqueName: \"kubernetes.io/projected/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735-kube-api-access-rvh2t\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:19.889186 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.889160 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.889257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.889237 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fa3225-0006-485d-a7de-59dd408b6f99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.889700 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.889672 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.889811 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.889712 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea59b41-386c-489e-811c-01f107d9458f-kube-api-access-sj8gr" pod "3ea59b41-386c-489e-811c-01f107d9458f" (UID: "3ea59b41-386c-489e-811c-01f107d9458f"). InnerVolumeSpecName "kube-api-access-sj8gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:19.889884 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.889808 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.889953 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.889931 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.891422 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.891398 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.891771 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.891751 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.892901 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.892874 6234 generic.go:356] "Generic (PLEG): container finished" podID="3ea59b41-386c-489e-811c-01f107d9458f" containerID="6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db" exitCode=0 Nov 03 22:45:19.892978 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.892920 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" Nov 03 22:45:19.892978 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.892959 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" event={"ID":"3ea59b41-386c-489e-811c-01f107d9458f","Type":"ContainerDied","Data":"6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db"} Nov 03 22:45:19.893080 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.893036 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss" event={"ID":"3ea59b41-386c-489e-811c-01f107d9458f","Type":"ContainerDied","Data":"d873029a57d6cb94fbc5a2ce0364035b699e515385f78feae13613c18944e3ad"} Nov 03 22:45:19.893080 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.893059 6234 scope.go:122] "RemoveContainer" containerID="6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db" Nov 03 22:45:19.894610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.894583 6234 generic.go:356] "Generic (PLEG): container finished" podID="5dc0ea6a-242f-44a2-aa04-e6b2c3b22735" containerID="bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d" exitCode=0 Nov 03 22:45:19.894700 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.894658 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" Nov 03 22:45:19.894700 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.894671 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" event={"ID":"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735","Type":"ContainerDied","Data":"bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d"} Nov 03 22:45:19.894811 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.894721 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g" event={"ID":"5dc0ea6a-242f-44a2-aa04-e6b2c3b22735","Type":"ContainerDied","Data":"a0d203bd220fc0c1018e408b8ec1a6233ef9ad68d38065ddf95c01ea77ae0ecf"} Nov 03 22:45:19.905219 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.905194 6234 scope.go:122] "RemoveContainer" containerID="6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db" Nov 03 22:45:19.905690 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:19.905555 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db\": container with ID starting with 6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db not found: ID does not exist" containerID="6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db" Nov 03 22:45:19.905690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.905589 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db"} err="failed to get container status \"6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db\": rpc error: code = NotFound desc = could not find container \"6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db\": container with ID starting with 6884be6e91ae39d39857901abc85809e72550340a06983d16e9e8975d0f2f9db not found: ID does not exist" Nov 03 22:45:19.905690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.905614 6234 scope.go:122] "RemoveContainer" containerID="bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d" Nov 03 22:45:19.914647 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.914630 6234 scope.go:122] "RemoveContainer" containerID="bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d" Nov 03 22:45:19.914901 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:19.914883 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d\": container with ID starting with bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d not found: ID does not exist" containerID="bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d" Nov 03 22:45:19.914949 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.914907 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d"} err="failed to get container status \"bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d\": rpc error: code = NotFound desc = could not find container \"bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d\": container with ID starting with bbaa4e876e9be4bbdaba4971dcf06dad565db5003610d38f349f9811ef90780d not found: ID does not exist" Nov 03 22:45:19.942621 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.942598 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l994b\" (UniqueName: \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-kube-api-access-l994b\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.942723 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.942700 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-j5xlv\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.968739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.968659 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:19.990062 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:19.990033 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj8gr\" (UniqueName: \"kubernetes.io/projected/3ea59b41-386c-489e-811c-01f107d9458f-kube-api-access-sj8gr\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:20.072307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.072276 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss"] Nov 03 22:45:20.080643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.078752 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3378fd-6455b77975-cgqss"] Nov 03 22:45:20.180645 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.180618 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g"] Nov 03 22:45:20.183709 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.183685 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3378fd-b4659cc4b-rxj6g"] Nov 03 22:45:20.407817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.407783 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv"] Nov 03 22:45:20.412488 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:45:20.412457 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fa3225_0006_485d_a7de_59dd408b6f99.slice/crio-31e89243dd18794f6b568fdb9bcbe572259f9a14ba72c1c4d78537b785ef938b WatchSource:0}: Error finding container 31e89243dd18794f6b568fdb9bcbe572259f9a14ba72c1c4d78537b785ef938b: Status 404 returned error can't find the container with id 31e89243dd18794f6b568fdb9bcbe572259f9a14ba72c1c4d78537b785ef938b Nov 03 22:45:20.415333 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.415300 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:20.415413 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.415369 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:20.415413 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.415402 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:20.701894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.701866 6234 scope.go:122] "RemoveContainer" containerID="70ad3ff6974ec1e2c5da87a7ee6435a82ce2ca6734f908eb3b588bd3347e7fea" Nov 03 22:45:20.707044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.707023 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea59b41-386c-489e-811c-01f107d9458f" path="/var/lib/kubelet/pods/3ea59b41-386c-489e-811c-01f107d9458f/volumes" Nov 03 22:45:20.707415 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.707401 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc0ea6a-242f-44a2-aa04-e6b2c3b22735" path="/var/lib/kubelet/pods/5dc0ea6a-242f-44a2-aa04-e6b2c3b22735/volumes" Nov 03 22:45:20.908536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.903166 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" event={"ID":"a0fa3225-0006-485d-a7de-59dd408b6f99","Type":"ContainerStarted","Data":"36c3dea3aa5eea7967ef8302e7a6a14265e7cf55aa6d512b7f4d3bc6add320f1"} Nov 03 22:45:20.908536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.903228 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" event={"ID":"a0fa3225-0006-485d-a7de-59dd408b6f99","Type":"ContainerStarted","Data":"31e89243dd18794f6b568fdb9bcbe572259f9a14ba72c1c4d78537b785ef938b"} Nov 03 22:45:20.908536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.906391 6234 generic.go:356] "Generic (PLEG): container finished" podID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerID="c435134b386086badc4bdaa8614412b37bf705dff1c01007bbec42409e81a1a9" exitCode=60 Nov 03 22:45:20.908536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.906464 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" event={"ID":"2791a167-71b5-4ee0-b76d-1e28f47cf03f","Type":"ContainerDied","Data":"c435134b386086badc4bdaa8614412b37bf705dff1c01007bbec42409e81a1a9"} Nov 03 22:45:20.908536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.906556 6234 scope.go:122] "RemoveContainer" containerID="e9ef60efc6fa35ee65f8845664a787ec2030ef29210ab2a67ca5b6eae747daab" Nov 03 22:45:20.908536 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.906865 6234 scope.go:122] "RemoveContainer" containerID="c435134b386086badc4bdaa8614412b37bf705dff1c01007bbec42409e81a1a9" Nov 03 22:45:20.908536 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:20.907129 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr_odh-ai-gateway-infra(2791a167-71b5-4ee0-b76d-1e28f47cf03f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" Nov 03 22:45:20.969696 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:20.969641 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:21.013871 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:21.013822 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" podStartSLOduration=2.013807429 podStartE2EDuration="2.013807429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:21.011568392 +0000 UTC m=+2240.423754411" watchObservedRunningTime="2026-11-03 22:45:21.013807429 +0000 UTC m=+2240.425993451" Nov 03 22:45:21.915106 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:21.915074 6234 generic.go:356] "Generic (PLEG): container finished" podID="3a392641-2369-47eb-9d75-e7316cf47589" containerID="eccd427531cd9dedce9f204d86b238c6770909a53e5de0528a2741570c37035d" exitCode=60 Nov 03 22:45:21.915623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:21.915149 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" event={"ID":"3a392641-2369-47eb-9d75-e7316cf47589","Type":"ContainerDied","Data":"eccd427531cd9dedce9f204d86b238c6770909a53e5de0528a2741570c37035d"} Nov 03 22:45:21.915623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:21.915213 6234 scope.go:122] "RemoveContainer" containerID="70ad3ff6974ec1e2c5da87a7ee6435a82ce2ca6734f908eb3b588bd3347e7fea" Nov 03 22:45:21.915711 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:21.915681 6234 scope.go:122] "RemoveContainer" containerID="eccd427531cd9dedce9f204d86b238c6770909a53e5de0528a2741570c37035d" Nov 03 22:45:21.916092 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:21.916065 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q_odh-ai-gateway-infra(3a392641-2369-47eb-9d75-e7316cf47589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" podUID="3a392641-2369-47eb-9d75-e7316cf47589" Nov 03 22:45:21.971124 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:21.971092 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:45:21.971289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:21.971139 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:45:22.131687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.131591 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr"] Nov 03 22:45:22.383488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.383428 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" Nov 03 22:45:22.530279 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.530243 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2791a167-71b5-4ee0-b76d-1e28f47cf03f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2791a167-71b5-4ee0-b76d-1e28f47cf03f-maas-api-service-ca\") pod \"2791a167-71b5-4ee0-b76d-1e28f47cf03f\" (UID: \"2791a167-71b5-4ee0-b76d-1e28f47cf03f\") " Nov 03 22:45:22.530478 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.530310 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2791a167-71b5-4ee0-b76d-1e28f47cf03f-kube-api-access-vl6sx\" (UniqueName: \"kubernetes.io/projected/2791a167-71b5-4ee0-b76d-1e28f47cf03f-kube-api-access-vl6sx\") pod \"2791a167-71b5-4ee0-b76d-1e28f47cf03f\" (UID: \"2791a167-71b5-4ee0-b76d-1e28f47cf03f\") " Nov 03 22:45:22.530982 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.530950 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2791a167-71b5-4ee0-b76d-1e28f47cf03f-maas-api-service-ca" pod "2791a167-71b5-4ee0-b76d-1e28f47cf03f" (UID: "2791a167-71b5-4ee0-b76d-1e28f47cf03f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:22.532806 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.532776 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2791a167-71b5-4ee0-b76d-1e28f47cf03f-kube-api-access-vl6sx" pod "2791a167-71b5-4ee0-b76d-1e28f47cf03f" (UID: "2791a167-71b5-4ee0-b76d-1e28f47cf03f"). InnerVolumeSpecName "kube-api-access-vl6sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:22.632113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.632078 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2791a167-71b5-4ee0-b76d-1e28f47cf03f-maas-api-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:22.632113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.632106 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl6sx\" (UniqueName: \"kubernetes.io/projected/2791a167-71b5-4ee0-b76d-1e28f47cf03f-kube-api-access-vl6sx\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:22.709939 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.709907 6234 scope.go:122] "RemoveContainer" containerID="9a429924f3f8c49ea84091e2265c36ee11f96d33af90675882bace54ac3957ea" Nov 03 22:45:22.927229 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.927203 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" Nov 03 22:45:22.927686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.927233 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr" event={"ID":"2791a167-71b5-4ee0-b76d-1e28f47cf03f","Type":"ContainerDied","Data":"00e2b9005c37c557e44f43640f54adb8200d885fe3de99e46ebbdda6180dfefb"} Nov 03 22:45:22.927686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.927304 6234 scope.go:122] "RemoveContainer" containerID="c435134b386086badc4bdaa8614412b37bf705dff1c01007bbec42409e81a1a9" Nov 03 22:45:22.969453 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.969371 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:45:22.969453 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:22.969420 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:45:23.032432 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.032400 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q"] Nov 03 22:45:23.082078 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.082044 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr"] Nov 03 22:45:23.089262 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.089226 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895765-zg9cr"] Nov 03 22:45:23.270737 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.270711 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" Nov 03 22:45:23.366137 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.366097 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a392641-2369-47eb-9d75-e7316cf47589-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a392641-2369-47eb-9d75-e7316cf47589-maas-api-service-ca\") pod \"3a392641-2369-47eb-9d75-e7316cf47589\" (UID: \"3a392641-2369-47eb-9d75-e7316cf47589\") " Nov 03 22:45:23.366377 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.366358 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a392641-2369-47eb-9d75-e7316cf47589-kube-api-access-xlh9b\" (UniqueName: \"kubernetes.io/projected/3a392641-2369-47eb-9d75-e7316cf47589-kube-api-access-xlh9b\") pod \"3a392641-2369-47eb-9d75-e7316cf47589\" (UID: \"3a392641-2369-47eb-9d75-e7316cf47589\") " Nov 03 22:45:23.366613 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.366584 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a392641-2369-47eb-9d75-e7316cf47589-maas-api-service-ca" pod "3a392641-2369-47eb-9d75-e7316cf47589" (UID: "3a392641-2369-47eb-9d75-e7316cf47589"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:23.366973 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.366935 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a392641-2369-47eb-9d75-e7316cf47589-maas-api-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.368658 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.368631 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a392641-2369-47eb-9d75-e7316cf47589-kube-api-access-xlh9b" pod "3a392641-2369-47eb-9d75-e7316cf47589" (UID: "3a392641-2369-47eb-9d75-e7316cf47589"). InnerVolumeSpecName "kube-api-access-xlh9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:23.468257 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.468210 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlh9b\" (UniqueName: \"kubernetes.io/projected/3a392641-2369-47eb-9d75-e7316cf47589-kube-api-access-xlh9b\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.936507 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.936462 6234 generic.go:356] "Generic (PLEG): container finished" podID="48d85139-0754-4092-8367-75e4dc27ee8f" containerID="5a4a69cc7b5f776c5a992147e5f9eec6302e75721be312f416307474ad3265f9" exitCode=60 Nov 03 22:45:23.936988 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.936522 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" event={"ID":"48d85139-0754-4092-8367-75e4dc27ee8f","Type":"ContainerDied","Data":"5a4a69cc7b5f776c5a992147e5f9eec6302e75721be312f416307474ad3265f9"} Nov 03 22:45:23.936988 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.936584 6234 scope.go:122] "RemoveContainer" containerID="9a429924f3f8c49ea84091e2265c36ee11f96d33af90675882bace54ac3957ea" Nov 03 22:45:23.936988 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.936944 6234 scope.go:122] "RemoveContainer" containerID="5a4a69cc7b5f776c5a992147e5f9eec6302e75721be312f416307474ad3265f9" Nov 03 22:45:23.937280 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:23.937249 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj_odh-ai-gateway-infra(48d85139-0754-4092-8367-75e4dc27ee8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" Nov 03 22:45:23.938113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.938096 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" Nov 03 22:45:23.938202 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.938098 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q" event={"ID":"3a392641-2369-47eb-9d75-e7316cf47589","Type":"ContainerDied","Data":"576b0ee426d0eba603549332fb19f3435bf5921f6addccd929e0fb80bd50534a"} Nov 03 22:45:23.972716 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.972679 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.972934 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.972765 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:45:23.981155 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:23.981135 6234 scope.go:122] "RemoveContainer" containerID="eccd427531cd9dedce9f204d86b238c6770909a53e5de0528a2741570c37035d" Nov 03 22:45:24.203518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:24.200050 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q"] Nov 03 22:45:24.237529 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:24.236870 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895765-pnv7q"] Nov 03 22:45:24.706049 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:24.706017 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" path="/var/lib/kubelet/pods/2791a167-71b5-4ee0-b76d-1e28f47cf03f/volumes" Nov 03 22:45:24.706637 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:24.706620 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a392641-2369-47eb-9d75-e7316cf47589" path="/var/lib/kubelet/pods/3a392641-2369-47eb-9d75-e7316cf47589/volumes" Nov 03 22:45:24.977569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:24.969096 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:45:24.977569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:24.969164 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:45:25.066673 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.066639 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj"] Nov 03 22:45:25.274207 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.274132 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" Nov 03 22:45:25.315650 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.315620 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d85139-0754-4092-8367-75e4dc27ee8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d85139-0754-4092-8367-75e4dc27ee8f-maas-api-service-ca\") pod \"48d85139-0754-4092-8367-75e4dc27ee8f\" (UID: \"48d85139-0754-4092-8367-75e4dc27ee8f\") " Nov 03 22:45:25.315849 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.315805 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d85139-0754-4092-8367-75e4dc27ee8f-kube-api-access-rlxdr\" (UniqueName: \"kubernetes.io/projected/48d85139-0754-4092-8367-75e4dc27ee8f-kube-api-access-rlxdr\") pod \"48d85139-0754-4092-8367-75e4dc27ee8f\" (UID: \"48d85139-0754-4092-8367-75e4dc27ee8f\") " Nov 03 22:45:25.316050 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.316024 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d85139-0754-4092-8367-75e4dc27ee8f-maas-api-service-ca" pod "48d85139-0754-4092-8367-75e4dc27ee8f" (UID: "48d85139-0754-4092-8367-75e4dc27ee8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:25.316183 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.316167 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d85139-0754-4092-8367-75e4dc27ee8f-maas-api-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:25.317815 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.317787 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d85139-0754-4092-8367-75e4dc27ee8f-kube-api-access-rlxdr" pod "48d85139-0754-4092-8367-75e4dc27ee8f" (UID: "48d85139-0754-4092-8367-75e4dc27ee8f"). InnerVolumeSpecName "kube-api-access-rlxdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:25.462519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.446729 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlxdr\" (UniqueName: \"kubernetes.io/projected/48d85139-0754-4092-8367-75e4dc27ee8f-kube-api-access-rlxdr\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:25.971663 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.971629 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:45:25.971864 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.971678 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:45:25.995041 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.995003 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" event={"ID":"48d85139-0754-4092-8367-75e4dc27ee8f","Type":"ContainerDied","Data":"ba501f5954d5f92513b502d4a1888d441562a60516da025db49f9e5a79ecd2f5"} Nov 03 22:45:25.995041 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.995030 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj" Nov 03 22:45:25.995677 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:25.995077 6234 scope.go:122] "RemoveContainer" containerID="5a4a69cc7b5f776c5a992147e5f9eec6302e75721be312f416307474ad3265f9" Nov 03 22:45:26.166691 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:26.166657 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj"] Nov 03 22:45:26.169837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:26.169810 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6a6b7f-29895765-h76jj"] Nov 03 22:45:26.706056 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:26.706018 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" path="/var/lib/kubelet/pods/48d85139-0754-4092-8367-75e4dc27ee8f/volumes" Nov 03 22:45:26.969316 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:26.969220 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.969316 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:26.969271 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:45:27.970004 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:27.969968 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:45:27.970533 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:27.970030 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 22:45:29.068939 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:29.068904 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:45:29.440561 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:29.440531 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:29.440982 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:29.440960 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:29.442088 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:29.442063 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:45:37.480999 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.480954 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p"] Nov 03 22:45:37.481915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.481889 6234 cpu_manager.go:414] "containerMap: removing container" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerName="cleanup" Nov 03 22:45:37.481915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.481916 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerName="cleanup" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.481931 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3ea59b41-386c-489e-811c-01f107d9458f" containerName="payload-pre-processing" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.481940 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea59b41-386c-489e-811c-01f107d9458f" containerName="payload-pre-processing" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.481963 6234 cpu_manager.go:414] "containerMap: removing container" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" containerName="cleanup" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.481971 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" containerName="cleanup" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.481996 6234 cpu_manager.go:414] "containerMap: removing container" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerName="cleanup" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482004 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerName="cleanup" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482018 6234 cpu_manager.go:414] "containerMap: removing container" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" containerName="cleanup" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482026 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" containerName="cleanup" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482038 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3a392641-2369-47eb-9d75-e7316cf47589" containerName="cleanup" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482046 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a392641-2369-47eb-9d75-e7316cf47589" containerName="cleanup" Nov 03 22:45:37.482059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482057 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3a392641-2369-47eb-9d75-e7316cf47589" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482066 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a392641-2369-47eb-9d75-e7316cf47589" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482083 6234 cpu_manager.go:414] "containerMap: removing container" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482090 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482119 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3a392641-2369-47eb-9d75-e7316cf47589" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482126 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a392641-2369-47eb-9d75-e7316cf47589" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482138 6234 cpu_manager.go:414] "containerMap: removing container" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482145 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482164 6234 cpu_manager.go:414] "containerMap: removing container" podUID="5dc0ea6a-242f-44a2-aa04-e6b2c3b22735" containerName="payload-processing" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482179 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc0ea6a-242f-44a2-aa04-e6b2c3b22735" containerName="payload-processing" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482335 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482348 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea59b41-386c-489e-811c-01f107d9458f" containerName="payload-pre-processing" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482361 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482373 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482382 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a392641-2369-47eb-9d75-e7316cf47589" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482390 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a392641-2369-47eb-9d75-e7316cf47589" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482399 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482411 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a392641-2369-47eb-9d75-e7316cf47589" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482421 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d85139-0754-4092-8367-75e4dc27ee8f" containerName="cleanup" Nov 03 22:45:37.482572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.482432 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc0ea6a-242f-44a2-aa04-e6b2c3b22735" containerName="payload-processing" Nov 03 22:45:37.492029 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.492007 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:45:37.567517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.565747 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p"] Nov 03 22:45:37.601516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.595819 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-295682\"" Nov 03 22:45:37.601516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.595976 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-295682-dockercfg-w2472\"" Nov 03 22:45:37.686518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.685430 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2gcr\" (UniqueName: \"kubernetes.io/projected/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-kube-api-access-l2gcr\") pod \"payload-processing-e2e-api-b-295682-7d66f469b-bdf7p\" (UID: \"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:45:37.686518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.685515 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-295682-7d66f469b-bdf7p\" (UID: \"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:45:37.720081 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.720044 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw"] Nov 03 22:45:37.725578 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.725549 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="2791a167-71b5-4ee0-b76d-1e28f47cf03f" containerName="cleanup" Nov 03 22:45:37.737039 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.736787 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:45:37.787270 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.786915 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2gcr\" (UniqueName: \"kubernetes.io/projected/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-kube-api-access-l2gcr\") pod \"payload-processing-e2e-api-b-295682-7d66f469b-bdf7p\" (UID: \"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:45:37.787598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.787573 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-295682-7d66f469b-bdf7p\" (UID: \"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:45:37.788942 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.788809 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-295682-7d66f469b-bdf7p\" (UID: \"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:45:37.789563 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.789541 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9fcdcb-09a9-4450-91a4-2c243670093a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw\" (UID: \"fd9fcdcb-09a9-4450-91a4-2c243670093a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:45:37.789785 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.789766 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpgw2\" (UniqueName: \"kubernetes.io/projected/fd9fcdcb-09a9-4450-91a4-2c243670093a-kube-api-access-tpgw2\") pod \"payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw\" (UID: \"fd9fcdcb-09a9-4450-91a4-2c243670093a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:45:37.818549 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.818403 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw"] Nov 03 22:45:37.900583 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.890881 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9fcdcb-09a9-4450-91a4-2c243670093a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw\" (UID: \"fd9fcdcb-09a9-4450-91a4-2c243670093a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:45:37.900885 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.900857 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpgw2\" (UniqueName: \"kubernetes.io/projected/fd9fcdcb-09a9-4450-91a4-2c243670093a-kube-api-access-tpgw2\") pod \"payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw\" (UID: \"fd9fcdcb-09a9-4450-91a4-2c243670093a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:45:37.901151 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.901122 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2gcr\" (UniqueName: \"kubernetes.io/projected/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-kube-api-access-l2gcr\") pod \"payload-processing-e2e-api-b-295682-7d66f469b-bdf7p\" (UID: \"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:45:37.901398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.893422 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9fcdcb-09a9-4450-91a4-2c243670093a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw\" (UID: \"fd9fcdcb-09a9-4450-91a4-2c243670093a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:45:37.956202 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:37.956167 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpgw2\" (UniqueName: \"kubernetes.io/projected/fd9fcdcb-09a9-4450-91a4-2c243670093a-kube-api-access-tpgw2\") pod \"payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw\" (UID: \"fd9fcdcb-09a9-4450-91a4-2c243670093a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:45:38.119585 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:38.119477 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:45:38.168063 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:38.167767 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:45:38.546315 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:38.546276 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw"] Nov 03 22:45:38.550946 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:45:38.550917 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9fcdcb_09a9_4450_91a4_2c243670093a.slice/crio-06cd31f83f1dbc14a90129105aeb8e8fbc159d53f2dae83a27d597cd9aa256de WatchSource:0}: Error finding container 06cd31f83f1dbc14a90129105aeb8e8fbc159d53f2dae83a27d597cd9aa256de: Status 404 returned error can't find the container with id 06cd31f83f1dbc14a90129105aeb8e8fbc159d53f2dae83a27d597cd9aa256de Nov 03 22:45:38.600266 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:38.600145 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p"] Nov 03 22:45:38.605710 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:45:38.605683 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f8c633_0135_48f9_b2f5_3d2a8cc06b3f.slice/crio-1f0f970d112c23257ccad982e79886c7eaaddcf49930a01985f013a6ccfaf634 WatchSource:0}: Error finding container 1f0f970d112c23257ccad982e79886c7eaaddcf49930a01985f013a6ccfaf634: Status 404 returned error can't find the container with id 1f0f970d112c23257ccad982e79886c7eaaddcf49930a01985f013a6ccfaf634 Nov 03 22:45:39.137955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:39.137912 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" event={"ID":"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f","Type":"ContainerStarted","Data":"8692bae8a9516d341b8aef969fed39a98aa76ccd84571c715b82522df8807e65"} Nov 03 22:45:39.138156 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:39.137980 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" event={"ID":"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f","Type":"ContainerStarted","Data":"1f0f970d112c23257ccad982e79886c7eaaddcf49930a01985f013a6ccfaf634"} Nov 03 22:45:39.138156 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:39.138086 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:45:39.139811 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:39.139782 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" event={"ID":"fd9fcdcb-09a9-4450-91a4-2c243670093a","Type":"ContainerStarted","Data":"dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702"} Nov 03 22:45:39.139959 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:39.139830 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" event={"ID":"fd9fcdcb-09a9-4450-91a4-2c243670093a","Type":"ContainerStarted","Data":"06cd31f83f1dbc14a90129105aeb8e8fbc159d53f2dae83a27d597cd9aa256de"} Nov 03 22:45:39.139959 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:39.139854 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:45:39.292686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:39.292635 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" podStartSLOduration=2.292619896 podStartE2EDuration="2.292619896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:39.245614432 +0000 UTC m=+2258.657800461" watchObservedRunningTime="2026-11-03 22:45:39.292619896 +0000 UTC m=+2258.704805913" Nov 03 22:45:39.355310 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:39.355161 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" podStartSLOduration=2.355143224 podStartE2EDuration="2.355143224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:39.350992096 +0000 UTC m=+2258.763178117" watchObservedRunningTime="2026-11-03 22:45:39.355143224 +0000 UTC m=+2258.767329241" Nov 03 22:45:50.156737 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.156656 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:45:50.157402 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.157377 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:45:50.324768 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.324734 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn"] Nov 03 22:45:50.329160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.329132 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.397551 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.391409 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn"] Nov 03 22:45:50.426858 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.426773 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-472b46-openshift-default-dockercfg-cxqkb\"" Nov 03 22:45:50.506267 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.500423 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-data\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.506267 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.500540 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-envoy\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.506267 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.500682 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-certs\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.506267 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.500711 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e787c8-9530-4516-ac62-2de3e352fdab-istiod-ca-cert\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.527577 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.523761 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-token\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.527577 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.523830 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-credential-socket\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.527577 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.523907 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-socket\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.527577 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.523977 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-podinfo\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.527577 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.524025 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjx79\" (UniqueName: \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-kube-api-access-bjx79\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.626185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.626127 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-envoy\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.626374 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.626357 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-certs\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.626436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.626387 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e787c8-9530-4516-ac62-2de3e352fdab-istiod-ca-cert\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.626515 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.626485 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-token\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.626582 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.626532 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-credential-socket\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.626582 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.626572 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-socket\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.626687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.626618 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-podinfo\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.626687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.626661 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjx79\" (UniqueName: \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-kube-api-access-bjx79\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.626794 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.626715 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-data\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.627018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.626993 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-socket\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.627119 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.627074 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-data\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.627408 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.627360 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-credential-socket\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.627758 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.627733 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e787c8-9530-4516-ac62-2de3e352fdab-istiod-ca-cert\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.627908 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.627860 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-certs\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.630933 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.629226 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-envoy\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.630933 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.630896 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-podinfo\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.680075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.680009 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-token\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.680217 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.680160 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjx79\" (UniqueName: \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-kube-api-access-bjx79\") pod \"e2e-api-b-472b46-openshift-default-5696968f69-wkvhn\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:50.943168 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:50.943139 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:51.373886 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:51.373802 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn"] Nov 03 22:45:51.378542 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:45:51.378513 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e787c8_9530_4516_ac62_2de3e352fdab.slice/crio-2e7e3d99d32481eafaff95b368390ab93ca310b40e3ae8bad1f55dacd5d12bbe WatchSource:0}: Error finding container 2e7e3d99d32481eafaff95b368390ab93ca310b40e3ae8bad1f55dacd5d12bbe: Status 404 returned error can't find the container with id 2e7e3d99d32481eafaff95b368390ab93ca310b40e3ae8bad1f55dacd5d12bbe Nov 03 22:45:51.381245 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:51.381215 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:51.381339 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:51.381278 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:51.381339 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:51.381319 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:52.237086 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:52.237050 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" event={"ID":"d8e787c8-9530-4516-ac62-2de3e352fdab","Type":"ContainerStarted","Data":"186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07"} Nov 03 22:45:52.237261 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:52.237100 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" event={"ID":"d8e787c8-9530-4516-ac62-2de3e352fdab","Type":"ContainerStarted","Data":"2e7e3d99d32481eafaff95b368390ab93ca310b40e3ae8bad1f55dacd5d12bbe"} Nov 03 22:45:52.393569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:52.393510 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" podStartSLOduration=2.393478607 podStartE2EDuration="2.393478607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:52.392585267 +0000 UTC m=+2271.804771285" watchObservedRunningTime="2026-11-03 22:45:52.393478607 +0000 UTC m=+2271.805664626" Nov 03 22:45:52.944193 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:52.944157 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:52.950041 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:52.950013 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:53.243222 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:53.243129 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:53.244365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:53.244341 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:45:57.457103 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.457028 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l"] Nov 03 22:45:57.461317 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.461297 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.524577 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.524546 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.524731 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.524585 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.524731 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.524612 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.525184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.524726 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.525184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.524829 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2gcw\" (UniqueName: \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-kube-api-access-j2gcw\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.525184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.524869 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.525184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.525012 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.525184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.525083 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.525184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.525115 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.626165 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626133 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.626346 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626186 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.626346 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626209 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.626346 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626232 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.626346 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626289 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2gcw\" (UniqueName: \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-kube-api-access-j2gcw\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.626346 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626336 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.626637 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626398 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.626637 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626454 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.626637 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626477 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.626904 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626875 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.627019 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626878 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.627019 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626967 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.627019 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626978 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.627019 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.626969 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.628807 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.628778 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.628973 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.628956 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.651126 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.651091 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l"] Nov 03 22:45:57.679452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.679414 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.679650 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.679631 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2gcw\" (UniqueName: \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-kube-api-access-j2gcw\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:57.776333 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:57.776243 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:58.205882 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:58.205845 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l"] Nov 03 22:45:58.305313 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:58.305266 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:58.305569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:58.305419 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:58.305569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:58.305465 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:58.316919 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:58.316881 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" event={"ID":"ec3815cd-a626-41e9-9a6f-d5a271b218d5","Type":"ContainerStarted","Data":"df3118a4c3e090789efe7e3bad69002609a08696721a21dbabf310e9fc658491"} Nov 03 22:45:58.997504 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:45:58.997454 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:45:59.325641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:59.325550 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" event={"ID":"ec3815cd-a626-41e9-9a6f-d5a271b218d5","Type":"ContainerStarted","Data":"82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054"} Nov 03 22:45:59.483085 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:59.483023 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" podStartSLOduration=2.483004276 podStartE2EDuration="2.483004276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:59.481798724 +0000 UTC m=+2278.893984742" watchObservedRunningTime="2026-11-03 22:45:59.483004276 +0000 UTC m=+2278.895190294" Nov 03 22:45:59.776722 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:59.776692 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:45:59.778318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:59.778290 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:45:59.778452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:45:59.778337 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 22:46:00.778000 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:00.777970 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:46:00.778474 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:00.778451 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 22:46:01.776590 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:01.776552 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:46:01.776761 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:01.776615 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 22:46:02.777139 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:02.777096 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:46:02.777559 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:02.777153 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 22:46:03.777223 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:03.777189 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:46:03.777648 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:03.777244 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 22:46:04.777211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:04.777175 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:46:04.777379 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:04.777245 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 22:46:05.823868 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:05.823831 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:46:05.824361 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:05.824134 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:46:05.824930 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:05.824905 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:46:16.945466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:16.945424 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:46:28.997684 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:46:28.997643 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:46:44.546229 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:44.546189 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m"] Nov 03 22:46:44.546788 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:44.546435 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" containerID="cri-o://dbf7a0ab5c3a40136ffe38e28eb967616fe177442018703b32f216ec56f4f397" gracePeriod=30 Nov 03 22:46:49.677066 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.677032 6234 generic.go:356] "Generic (PLEG): container finished" podID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerID="dbf7a0ab5c3a40136ffe38e28eb967616fe177442018703b32f216ec56f4f397" exitCode=0 Nov 03 22:46:49.677407 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.677097 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" event={"ID":"f07f28fd-b732-4a25-ac79-2381cea9a252","Type":"ContainerDied","Data":"dbf7a0ab5c3a40136ffe38e28eb967616fe177442018703b32f216ec56f4f397"} Nov 03 22:46:49.824775 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.824708 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:46:49.864421 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864393 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07f28fd-b732-4a25-ac79-2381cea9a252-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07f28fd-b732-4a25-ac79-2381cea9a252-istiod-ca-cert\") pod \"f07f28fd-b732-4a25-ac79-2381cea9a252\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " Nov 03 22:46:49.864593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864470 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-credential-socket\") pod \"f07f28fd-b732-4a25-ac79-2381cea9a252\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " Nov 03 22:46:49.864593 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864550 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-data\") pod \"f07f28fd-b732-4a25-ac79-2381cea9a252\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " Nov 03 22:46:49.864685 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864590 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-kube-api-access-4tc67\" (UniqueName: \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-kube-api-access-4tc67\") pod \"f07f28fd-b732-4a25-ac79-2381cea9a252\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " Nov 03 22:46:49.864685 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864633 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-envoy\") pod \"f07f28fd-b732-4a25-ac79-2381cea9a252\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " Nov 03 22:46:49.864763 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864709 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-socket\") pod \"f07f28fd-b732-4a25-ac79-2381cea9a252\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " Nov 03 22:46:49.864763 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864740 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-credential-socket" pod "f07f28fd-b732-4a25-ac79-2381cea9a252" (UID: "f07f28fd-b732-4a25-ac79-2381cea9a252"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:49.864865 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864806 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-certs\") pod \"f07f28fd-b732-4a25-ac79-2381cea9a252\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " Nov 03 22:46:49.864865 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864835 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-podinfo\") pod \"f07f28fd-b732-4a25-ac79-2381cea9a252\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " Nov 03 22:46:49.864865 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864845 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-data" pod "f07f28fd-b732-4a25-ac79-2381cea9a252" (UID: "f07f28fd-b732-4a25-ac79-2381cea9a252"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:49.865026 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864890 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07f28fd-b732-4a25-ac79-2381cea9a252-istiod-ca-cert" pod "f07f28fd-b732-4a25-ac79-2381cea9a252" (UID: "f07f28fd-b732-4a25-ac79-2381cea9a252"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:49.865026 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.864894 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-token\" (UniqueName: \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-token\") pod \"f07f28fd-b732-4a25-ac79-2381cea9a252\" (UID: \"f07f28fd-b732-4a25-ac79-2381cea9a252\") " Nov 03 22:46:49.865136 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.865078 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-socket" pod "f07f28fd-b732-4a25-ac79-2381cea9a252" (UID: "f07f28fd-b732-4a25-ac79-2381cea9a252"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:49.865136 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.865095 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-certs" pod "f07f28fd-b732-4a25-ac79-2381cea9a252" (UID: "f07f28fd-b732-4a25-ac79-2381cea9a252"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:49.865485 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.865463 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07f28fd-b732-4a25-ac79-2381cea9a252-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:49.865604 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.865491 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:49.865604 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.865535 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:49.865604 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.865549 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:49.865604 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.865563 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:49.866847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.866815 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-envoy" pod "f07f28fd-b732-4a25-ac79-2381cea9a252" (UID: "f07f28fd-b732-4a25-ac79-2381cea9a252"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:49.866947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.866910 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-kube-api-access-4tc67" pod "f07f28fd-b732-4a25-ac79-2381cea9a252" (UID: "f07f28fd-b732-4a25-ac79-2381cea9a252"). InnerVolumeSpecName "kube-api-access-4tc67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:49.867017 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.866948 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-podinfo" pod "f07f28fd-b732-4a25-ac79-2381cea9a252" (UID: "f07f28fd-b732-4a25-ac79-2381cea9a252"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:46:49.867075 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.867063 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-token" pod "f07f28fd-b732-4a25-ac79-2381cea9a252" (UID: "f07f28fd-b732-4a25-ac79-2381cea9a252"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:49.967401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.967371 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:49.967401 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.967402 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tc67\" (UniqueName: \"kubernetes.io/projected/f07f28fd-b732-4a25-ac79-2381cea9a252-kube-api-access-4tc67\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:49.967612 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.967413 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:49.967612 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:49.967422 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07f28fd-b732-4a25-ac79-2381cea9a252-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:50.684131 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:50.684098 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" event={"ID":"f07f28fd-b732-4a25-ac79-2381cea9a252","Type":"ContainerDied","Data":"4d70e1610ae8c598b13b87aededcb849170e73eab64f89f0870c29284bef2897"} Nov 03 22:46:50.684131 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:50.684126 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" Nov 03 22:46:50.684672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:50.684162 6234 scope.go:122] "RemoveContainer" containerID="dbf7a0ab5c3a40136ffe38e28eb967616fe177442018703b32f216ec56f4f397" Nov 03 22:46:50.810857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:50.810816 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:46:50.811040 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:50.810884 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:46:50.842996 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:50.842956 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m"] Nov 03 22:46:50.889557 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:50.889514 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-m5z8m"] Nov 03 22:46:52.706154 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:46:52.706120 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" path="/var/lib/kubelet/pods/f07f28fd-b732-4a25-ac79-2381cea9a252/volumes" Nov 03 22:46:59.003052 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:46:59.003012 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:47:02.818263 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:02.818227 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:47:28.997232 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:47:28.997151 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:47:32.457942 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.457864 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp"] Nov 03 22:47:32.460055 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.458820 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" podUID="5163c5b4-4805-4adc-9216-8958b17a2b6d" containerName="maas-api" containerID="cri-o://e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d" gracePeriod=30 Nov 03 22:47:32.813263 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.813236 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:47:32.851371 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.851341 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5163c5b4-4805-4adc-9216-8958b17a2b6d-kube-api-access-dw6s8\" (UniqueName: \"kubernetes.io/projected/5163c5b4-4805-4adc-9216-8958b17a2b6d-kube-api-access-dw6s8\") pod \"5163c5b4-4805-4adc-9216-8958b17a2b6d\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " Nov 03 22:47:32.851546 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.851479 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-metrics-certs\") pod \"5163c5b4-4805-4adc-9216-8958b17a2b6d\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " Nov 03 22:47:32.851719 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.851698 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5163c5b4-4805-4adc-9216-8958b17a2b6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5163c5b4-4805-4adc-9216-8958b17a2b6d-openshift-service-ca\") pod \"5163c5b4-4805-4adc-9216-8958b17a2b6d\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " Nov 03 22:47:32.851794 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.851733 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-maas-api-tls\") pod \"5163c5b4-4805-4adc-9216-8958b17a2b6d\" (UID: \"5163c5b4-4805-4adc-9216-8958b17a2b6d\") " Nov 03 22:47:32.852089 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.852061 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5163c5b4-4805-4adc-9216-8958b17a2b6d-openshift-service-ca" pod "5163c5b4-4805-4adc-9216-8958b17a2b6d" (UID: "5163c5b4-4805-4adc-9216-8958b17a2b6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:32.852440 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.852415 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5163c5b4-4805-4adc-9216-8958b17a2b6d-openshift-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:32.853675 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.853648 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-metrics-certs" pod "5163c5b4-4805-4adc-9216-8958b17a2b6d" (UID: "5163c5b4-4805-4adc-9216-8958b17a2b6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:32.853786 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.853695 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5163c5b4-4805-4adc-9216-8958b17a2b6d-kube-api-access-dw6s8" pod "5163c5b4-4805-4adc-9216-8958b17a2b6d" (UID: "5163c5b4-4805-4adc-9216-8958b17a2b6d"). InnerVolumeSpecName "kube-api-access-dw6s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:32.853868 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.853834 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-maas-api-tls" pod "5163c5b4-4805-4adc-9216-8958b17a2b6d" (UID: "5163c5b4-4805-4adc-9216-8958b17a2b6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:32.956518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.955630 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-metrics-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:32.956518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.955667 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5163c5b4-4805-4adc-9216-8958b17a2b6d-maas-api-tls\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:32.956518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.955683 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw6s8\" (UniqueName: \"kubernetes.io/projected/5163c5b4-4805-4adc-9216-8958b17a2b6d-kube-api-access-dw6s8\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:32.980519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.977621 6234 generic.go:356] "Generic (PLEG): container finished" podID="5163c5b4-4805-4adc-9216-8958b17a2b6d" containerID="e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d" exitCode=0 Nov 03 22:47:32.980519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.977780 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" event={"ID":"5163c5b4-4805-4adc-9216-8958b17a2b6d","Type":"ContainerDied","Data":"e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d"} Nov 03 22:47:32.980519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.977841 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" event={"ID":"5163c5b4-4805-4adc-9216-8958b17a2b6d","Type":"ContainerDied","Data":"c14c973a42aff988d8df9feb227a4479a3101ebac53c1987834a14cf76479025"} Nov 03 22:47:32.980519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.977874 6234 scope.go:122] "RemoveContainer" containerID="e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d" Nov 03 22:47:32.980519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:32.978043 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp" Nov 03 22:47:33.011054 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:33.010868 6234 scope.go:122] "RemoveContainer" containerID="e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d" Nov 03 22:47:33.011272 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:47:33.011240 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d\": container with ID starting with e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d not found: ID does not exist" containerID="e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d" Nov 03 22:47:33.011346 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:33.011289 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d"} err="failed to get container status \"e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d\": rpc error: code = NotFound desc = could not find container \"e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d\": container with ID starting with e8377d40f09daee90465767c0d1f14d638be63b794a81138930a54a998e98d4d not found: ID does not exist" Nov 03 22:47:33.163098 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:33.162985 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp"] Nov 03 22:47:33.165759 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:33.165734 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3378fd-f44d5bcc9-h6dsp"] Nov 03 22:47:33.762580 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:33.762548 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt"] Nov 03 22:47:33.763151 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:33.762731 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" podUID="228e0c72-3449-49d8-a3c5-d96d0a8ccacc" containerName="maas-api" containerID="cri-o://cb23c82aa301e2e3e6b5ba03b14f1787c5906890fc41b7ae917249233dd052bb" gracePeriod=30 Nov 03 22:47:34.029779 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.029642 6234 generic.go:356] "Generic (PLEG): container finished" podID="228e0c72-3449-49d8-a3c5-d96d0a8ccacc" containerID="cb23c82aa301e2e3e6b5ba03b14f1787c5906890fc41b7ae917249233dd052bb" exitCode=0 Nov 03 22:47:34.030160 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.030133 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" event={"ID":"228e0c72-3449-49d8-a3c5-d96d0a8ccacc","Type":"ContainerDied","Data":"cb23c82aa301e2e3e6b5ba03b14f1787c5906890fc41b7ae917249233dd052bb"} Nov 03 22:47:34.091178 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.091151 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:47:34.232017 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.231966 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-openshift-service-ca\") pod \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " Nov 03 22:47:34.232216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.232079 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-kube-api-access-62dlb\" (UniqueName: \"kubernetes.io/projected/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-kube-api-access-62dlb\") pod \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " Nov 03 22:47:34.232216 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.232185 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-metrics-certs\") pod \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " Nov 03 22:47:34.232544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.232518 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-maas-api-tls\") pod \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\" (UID: \"228e0c72-3449-49d8-a3c5-d96d0a8ccacc\") " Nov 03 22:47:34.236708 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.236676 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-maas-api-tls" pod "228e0c72-3449-49d8-a3c5-d96d0a8ccacc" (UID: "228e0c72-3449-49d8-a3c5-d96d0a8ccacc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:34.237055 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.237034 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-openshift-service-ca" pod "228e0c72-3449-49d8-a3c5-d96d0a8ccacc" (UID: "228e0c72-3449-49d8-a3c5-d96d0a8ccacc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:34.238916 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.238894 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-kube-api-access-62dlb" pod "228e0c72-3449-49d8-a3c5-d96d0a8ccacc" (UID: "228e0c72-3449-49d8-a3c5-d96d0a8ccacc"). InnerVolumeSpecName "kube-api-access-62dlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:34.246535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.242604 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-metrics-certs" pod "228e0c72-3449-49d8-a3c5-d96d0a8ccacc" (UID: "228e0c72-3449-49d8-a3c5-d96d0a8ccacc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:34.362562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.357600 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-maas-api-tls\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:34.362562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.357638 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-openshift-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:34.362562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.357654 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62dlb\" (UniqueName: \"kubernetes.io/projected/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-kube-api-access-62dlb\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:34.362562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.357669 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/228e0c72-3449-49d8-a3c5-d96d0a8ccacc-metrics-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:34.706437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:34.706395 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5163c5b4-4805-4adc-9216-8958b17a2b6d" path="/var/lib/kubelet/pods/5163c5b4-4805-4adc-9216-8958b17a2b6d/volumes" Nov 03 22:47:35.040398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:35.040325 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" Nov 03 22:47:35.040875 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:35.040322 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt" event={"ID":"228e0c72-3449-49d8-a3c5-d96d0a8ccacc","Type":"ContainerDied","Data":"201e4c52b5a16640aed394d36ce41c7939b59333c680e7c73ee2de51a0924901"} Nov 03 22:47:35.040875 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:35.040459 6234 scope.go:122] "RemoveContainer" containerID="cb23c82aa301e2e3e6b5ba03b14f1787c5906890fc41b7ae917249233dd052bb" Nov 03 22:47:35.224072 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:35.224036 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt"] Nov 03 22:47:35.267882 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:35.267846 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6a6b7f-5499d547d7-b6lbt"] Nov 03 22:47:36.706118 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:36.706075 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="228e0c72-3449-49d8-a3c5-d96d0a8ccacc" path="/var/lib/kubelet/pods/228e0c72-3449-49d8-a3c5-d96d0a8ccacc/volumes" Nov 03 22:47:41.360467 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:41.360433 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd"] Nov 03 22:47:41.360938 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:41.360656 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerName="istio-proxy" containerID="cri-o://e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700" gracePeriod=30 Nov 03 22:47:41.376009 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:41.375981 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l"] Nov 03 22:47:41.376223 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:41.376185 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerName="istio-proxy" containerID="cri-o://82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054" gracePeriod=30 Nov 03 22:47:42.453152 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:42.453109 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:47:42.453650 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:42.453184 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:47:43.301777 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:43.301743 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68"] Nov 03 22:47:43.301966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:43.301944 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" containerID="cri-o://2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69" gracePeriod=30 Nov 03 22:47:44.726184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:44.726131 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:47:46.274481 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.274446 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt"] Nov 03 22:47:46.275292 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.275256 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerName="istio-proxy" containerID="cri-o://65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361" gracePeriod=30 Nov 03 22:47:46.806847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.806486 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:47:46.812699 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.812677 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.948893 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-kube-api-access-j2gcw\" (UniqueName: \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-kube-api-access-j2gcw\") pod \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.948949 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-data\") pod \"e10c1781-8bc9-4711-88b6-8b406d29ec25\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.948978 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e10c1781-8bc9-4711-88b6-8b406d29ec25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10c1781-8bc9-4711-88b6-8b406d29ec25-istiod-ca-cert\") pod \"e10c1781-8bc9-4711-88b6-8b406d29ec25\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949026 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-socket\") pod \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949064 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-podinfo\") pod \"e10c1781-8bc9-4711-88b6-8b406d29ec25\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949096 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-certs\") pod \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949132 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-certs\") pod \"e10c1781-8bc9-4711-88b6-8b406d29ec25\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949189 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-data\") pod \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949245 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-envoy\") pod \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949283 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-podinfo\") pod \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949330 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-envoy\") pod \"e10c1781-8bc9-4711-88b6-8b406d29ec25\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949407 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-token\" (UniqueName: \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-token\") pod \"e10c1781-8bc9-4711-88b6-8b406d29ec25\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949447 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-token\" (UniqueName: \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-token\") pod \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949545 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-socket\") pod \"e10c1781-8bc9-4711-88b6-8b406d29ec25\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949580 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-kube-api-access-lhn8p\" (UniqueName: \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-kube-api-access-lhn8p\") pod \"e10c1781-8bc9-4711-88b6-8b406d29ec25\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949695 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istiod-ca-cert\") pod \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949794 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-credential-socket\") pod \"e10c1781-8bc9-4711-88b6-8b406d29ec25\" (UID: \"e10c1781-8bc9-4711-88b6-8b406d29ec25\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.949830 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-credential-socket\") pod \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\" (UID: \"ec3815cd-a626-41e9-9a6f-d5a271b218d5\") " Nov 03 22:47:46.958545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.950806 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-credential-socket" pod "ec3815cd-a626-41e9-9a6f-d5a271b218d5" (UID: "ec3815cd-a626-41e9-9a6f-d5a271b218d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.974388 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.974338 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10c1781-8bc9-4711-88b6-8b406d29ec25-istiod-ca-cert" pod "e10c1781-8bc9-4711-88b6-8b406d29ec25" (UID: "e10c1781-8bc9-4711-88b6-8b406d29ec25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:46.974611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.974588 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-data" pod "e10c1781-8bc9-4711-88b6-8b406d29ec25" (UID: "e10c1781-8bc9-4711-88b6-8b406d29ec25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.975135 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.974856 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-certs" pod "ec3815cd-a626-41e9-9a6f-d5a271b218d5" (UID: "ec3815cd-a626-41e9-9a6f-d5a271b218d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.975135 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.974854 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-socket" pod "e10c1781-8bc9-4711-88b6-8b406d29ec25" (UID: "e10c1781-8bc9-4711-88b6-8b406d29ec25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.975135 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.975060 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-certs" pod "e10c1781-8bc9-4711-88b6-8b406d29ec25" (UID: "e10c1781-8bc9-4711-88b6-8b406d29ec25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.975356 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.975234 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-data" pod "ec3815cd-a626-41e9-9a6f-d5a271b218d5" (UID: "ec3815cd-a626-41e9-9a6f-d5a271b218d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.975356 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.975344 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-socket" pod "ec3815cd-a626-41e9-9a6f-d5a271b218d5" (UID: "ec3815cd-a626-41e9-9a6f-d5a271b218d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.975967 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.975941 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istiod-ca-cert" pod "ec3815cd-a626-41e9-9a6f-d5a271b218d5" (UID: "ec3815cd-a626-41e9-9a6f-d5a271b218d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:46.976174 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.976135 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-credential-socket" pod "e10c1781-8bc9-4711-88b6-8b406d29ec25" (UID: "e10c1781-8bc9-4711-88b6-8b406d29ec25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.976618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.976573 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-kube-api-access-j2gcw" pod "ec3815cd-a626-41e9-9a6f-d5a271b218d5" (UID: "ec3815cd-a626-41e9-9a6f-d5a271b218d5"). InnerVolumeSpecName "kube-api-access-j2gcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.984609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.981581 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-podinfo" pod "ec3815cd-a626-41e9-9a6f-d5a271b218d5" (UID: "ec3815cd-a626-41e9-9a6f-d5a271b218d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:46.984969 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.984946 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-envoy" pod "ec3815cd-a626-41e9-9a6f-d5a271b218d5" (UID: "ec3815cd-a626-41e9-9a6f-d5a271b218d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.991687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.991658 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-envoy" pod "e10c1781-8bc9-4711-88b6-8b406d29ec25" (UID: "e10c1781-8bc9-4711-88b6-8b406d29ec25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.991806 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.991779 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-token" pod "ec3815cd-a626-41e9-9a6f-d5a271b218d5" (UID: "ec3815cd-a626-41e9-9a6f-d5a271b218d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.993060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.993031 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-kube-api-access-lhn8p" pod "e10c1781-8bc9-4711-88b6-8b406d29ec25" (UID: "e10c1781-8bc9-4711-88b6-8b406d29ec25"). InnerVolumeSpecName "kube-api-access-lhn8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.993336 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.993315 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-token" pod "e10c1781-8bc9-4711-88b6-8b406d29ec25" (UID: "e10c1781-8bc9-4711-88b6-8b406d29ec25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.993466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:46.993444 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-podinfo" pod "e10c1781-8bc9-4711-88b6-8b406d29ec25" (UID: "e10c1781-8bc9-4711-88b6-8b406d29ec25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053759 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053818 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053833 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2gcw\" (UniqueName: \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-kube-api-access-j2gcw\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053849 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053864 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10c1781-8bc9-4711-88b6-8b406d29ec25-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053878 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053892 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053906 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053919 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053933 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053947 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053963 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053978 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.053990 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.054005 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.054019 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10c1781-8bc9-4711-88b6-8b406d29ec25-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.054034 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhn8p\" (UniqueName: \"kubernetes.io/projected/e10c1781-8bc9-4711-88b6-8b406d29ec25-kube-api-access-lhn8p\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.055398 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.054050 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3815cd-a626-41e9-9a6f-d5a271b218d5-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:47.181020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.180980 6234 generic.go:356] "Generic (PLEG): container finished" podID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerID="82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054" exitCode=0 Nov 03 22:47:47.181334 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.181079 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" event={"ID":"ec3815cd-a626-41e9-9a6f-d5a271b218d5","Type":"ContainerDied","Data":"82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054"} Nov 03 22:47:47.181334 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.181125 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" event={"ID":"ec3815cd-a626-41e9-9a6f-d5a271b218d5","Type":"ContainerDied","Data":"df3118a4c3e090789efe7e3bad69002609a08696721a21dbabf310e9fc658491"} Nov 03 22:47:47.181334 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.181152 6234 scope.go:122] "RemoveContainer" containerID="82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054" Nov 03 22:47:47.181334 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.181301 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l" Nov 03 22:47:47.183704 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.183673 6234 generic.go:356] "Generic (PLEG): container finished" podID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerID="e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700" exitCode=0 Nov 03 22:47:47.183847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.183735 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" event={"ID":"e10c1781-8bc9-4711-88b6-8b406d29ec25","Type":"ContainerDied","Data":"e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700"} Nov 03 22:47:47.183847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.183747 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" Nov 03 22:47:47.183847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.183783 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd" event={"ID":"e10c1781-8bc9-4711-88b6-8b406d29ec25","Type":"ContainerDied","Data":"58dde70996a5bc0e035e90d0c559cdf4e6bea0ad838e33e8560c6acbc9ba5f9c"} Nov 03 22:47:47.194700 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.194680 6234 scope.go:122] "RemoveContainer" containerID="82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054" Nov 03 22:47:47.194982 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:47:47.194962 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054\": container with ID starting with 82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054 not found: ID does not exist" containerID="82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054" Nov 03 22:47:47.195076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.194998 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054"} err="failed to get container status \"82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054\": rpc error: code = NotFound desc = could not find container \"82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054\": container with ID starting with 82c846f5758d1944602c4bddfac7b6ce27e2f563e81016e2b3b7a5b661ac3054 not found: ID does not exist" Nov 03 22:47:47.195076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.195022 6234 scope.go:122] "RemoveContainer" containerID="e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700" Nov 03 22:47:47.205051 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.205027 6234 scope.go:122] "RemoveContainer" containerID="e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700" Nov 03 22:47:47.205364 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:47:47.205339 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700\": container with ID starting with e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700 not found: ID does not exist" containerID="e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700" Nov 03 22:47:47.205424 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.205373 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700"} err="failed to get container status \"e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700\": rpc error: code = NotFound desc = could not find container \"e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700\": container with ID starting with e5bdad92f9a0252f0117c833c115c05d2e3cba57ee611b31f37d0e976934a700 not found: ID does not exist" Nov 03 22:47:47.367857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.367767 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l"] Nov 03 22:47:47.382891 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.382865 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-skt9l"] Nov 03 22:47:47.486469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.486369 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd"] Nov 03 22:47:47.494650 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:47.494615 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-jglrd"] Nov 03 22:47:48.613575 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.613549 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:47:48.706069 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.706032 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" path="/var/lib/kubelet/pods/e10c1781-8bc9-4711-88b6-8b406d29ec25/volumes" Nov 03 22:47:48.706534 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.706520 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" path="/var/lib/kubelet/pods/ec3815cd-a626-41e9-9a6f-d5a271b218d5/volumes" Nov 03 22:47:48.799482 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.799447 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-podinfo\") pod \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " Nov 03 22:47:48.799705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.799556 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-data\") pod \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " Nov 03 22:47:48.799705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.799635 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-envoy\") pod \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " Nov 03 22:47:48.799705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.799701 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-certs\") pod \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " Nov 03 22:47:48.799895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.799722 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-token\" (UniqueName: \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-token\") pod \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " Nov 03 22:47:48.799895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.799754 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-socket\") pod \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " Nov 03 22:47:48.799895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.799850 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-kube-api-access-gt7zg\" (UniqueName: \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-kube-api-access-gt7zg\") pod \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " Nov 03 22:47:48.799895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.799873 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istiod-ca-cert\") pod \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " Nov 03 22:47:48.800109 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.799897 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-credential-socket\") pod \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\" (UID: \"d24285a4-1ad4-4d6b-96da-75f575d24b5e\") " Nov 03 22:47:48.800439 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.800405 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-data" pod "d24285a4-1ad4-4d6b-96da-75f575d24b5e" (UID: "d24285a4-1ad4-4d6b-96da-75f575d24b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.800594 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.800568 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-certs" pod "d24285a4-1ad4-4d6b-96da-75f575d24b5e" (UID: "d24285a4-1ad4-4d6b-96da-75f575d24b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.800746 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.800713 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.800746 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.800741 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.800983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.800949 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istiod-ca-cert" pod "d24285a4-1ad4-4d6b-96da-75f575d24b5e" (UID: "d24285a4-1ad4-4d6b-96da-75f575d24b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:48.801350 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.801325 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-credential-socket" pod "d24285a4-1ad4-4d6b-96da-75f575d24b5e" (UID: "d24285a4-1ad4-4d6b-96da-75f575d24b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.801591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.801568 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-socket" pod "d24285a4-1ad4-4d6b-96da-75f575d24b5e" (UID: "d24285a4-1ad4-4d6b-96da-75f575d24b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.803346 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.803314 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-podinfo" pod "d24285a4-1ad4-4d6b-96da-75f575d24b5e" (UID: "d24285a4-1ad4-4d6b-96da-75f575d24b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:48.803507 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.803464 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-token" pod "d24285a4-1ad4-4d6b-96da-75f575d24b5e" (UID: "d24285a4-1ad4-4d6b-96da-75f575d24b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:48.803625 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.803526 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-envoy" pod "d24285a4-1ad4-4d6b-96da-75f575d24b5e" (UID: "d24285a4-1ad4-4d6b-96da-75f575d24b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.803690 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.803621 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-kube-api-access-gt7zg" pod "d24285a4-1ad4-4d6b-96da-75f575d24b5e" (UID: "d24285a4-1ad4-4d6b-96da-75f575d24b5e"). InnerVolumeSpecName "kube-api-access-gt7zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:48.901817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.901729 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt7zg\" (UniqueName: \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-kube-api-access-gt7zg\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.901817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.901763 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.901817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.901773 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.901817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.901782 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.901817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.901791 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.901817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.901799 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24285a4-1ad4-4d6b-96da-75f575d24b5e-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.901817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:48.901808 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24285a4-1ad4-4d6b-96da-75f575d24b5e-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:49.202195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:49.202143 6234 generic.go:356] "Generic (PLEG): container finished" podID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerID="2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69" exitCode=0 Nov 03 22:47:49.202392 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:49.202199 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" Nov 03 22:47:49.202392 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:49.202221 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" event={"ID":"d24285a4-1ad4-4d6b-96da-75f575d24b5e","Type":"ContainerDied","Data":"2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69"} Nov 03 22:47:49.202392 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:49.202291 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68" event={"ID":"d24285a4-1ad4-4d6b-96da-75f575d24b5e","Type":"ContainerDied","Data":"4c93a5981cc7813b7124d2e46a114416a52aa48157101bc5e8ab62b6754a1126"} Nov 03 22:47:49.202392 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:49.202326 6234 scope.go:122] "RemoveContainer" containerID="2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69" Nov 03 22:47:49.215908 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:49.215888 6234 scope.go:122] "RemoveContainer" containerID="2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69" Nov 03 22:47:49.216173 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:47:49.216150 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69\": container with ID starting with 2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69 not found: ID does not exist" containerID="2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69" Nov 03 22:47:49.216256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:49.216179 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69"} err="failed to get container status \"2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69\": rpc error: code = NotFound desc = could not find container \"2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69\": container with ID starting with 2a17528027187b7c41fc4f6aecb3870050cd23a1898969b0ed13a122fb937b69 not found: ID does not exist" Nov 03 22:47:49.367888 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:49.367845 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68"] Nov 03 22:47:49.370251 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:49.370220 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-mlk68"] Nov 03 22:47:50.706023 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:50.705989 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" path="/var/lib/kubelet/pods/d24285a4-1ad4-4d6b-96da-75f575d24b5e/volumes" Nov 03 22:47:51.609151 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.609124 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:47:51.637577 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.637542 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-certs\") pod \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " Nov 03 22:47:51.637739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.637621 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-credential-socket\") pod \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " Nov 03 22:47:51.637739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.637676 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-data\") pod \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " Nov 03 22:47:51.637739 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.637725 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-kube-api-access-zmnhs\" (UniqueName: \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-kube-api-access-zmnhs\") pod \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " Nov 03 22:47:51.637952 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.637836 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-token\" (UniqueName: \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-token\") pod \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " Nov 03 22:47:51.637952 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.637858 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-certs" pod "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" (UID: "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:51.637952 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.637903 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-podinfo\") pod \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " Nov 03 22:47:51.638116 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.637946 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-credential-socket" pod "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" (UID: "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:51.638116 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.637968 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-socket\") pod \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " Nov 03 22:47:51.638116 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.638089 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-data" pod "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" (UID: "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:51.638276 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.638256 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-socket" pod "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" (UID: "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:51.638603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.638578 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:51.638603 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.638605 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:51.638793 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.638617 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:51.638793 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.638627 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:51.640531 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.640509 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-podinfo" pod "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" (UID: "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:51.640637 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.640567 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-token" pod "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" (UID: "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:51.640637 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.640593 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-kube-api-access-zmnhs" pod "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" (UID: "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3"). InnerVolumeSpecName "kube-api-access-zmnhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:51.739343 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.739311 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-envoy\") pod \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " Nov 03 22:47:51.739795 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.739364 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istiod-ca-cert\") pod \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\" (UID: \"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3\") " Nov 03 22:47:51.739795 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.739765 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istiod-ca-cert" pod "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" (UID: "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:51.739939 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.739909 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:51.739939 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.739933 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:51.740027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.739950 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:51.740027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.739964 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmnhs\" (UniqueName: \"kubernetes.io/projected/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-kube-api-access-zmnhs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:51.741220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.741200 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-envoy" pod "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" (UID: "7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:51.840929 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:51.840845 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:52.222912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:52.222875 6234 generic.go:356] "Generic (PLEG): container finished" podID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerID="65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361" exitCode=0 Nov 03 22:47:52.223065 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:52.222931 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" Nov 03 22:47:52.223065 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:52.222973 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" event={"ID":"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3","Type":"ContainerDied","Data":"65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361"} Nov 03 22:47:52.223065 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:52.223042 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt" event={"ID":"7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3","Type":"ContainerDied","Data":"199df3ebceb28bfd79b115308a047a037026f6e659b5d3c4e847520df3041aba"} Nov 03 22:47:52.223159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:52.223075 6234 scope.go:122] "RemoveContainer" containerID="65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361" Nov 03 22:47:52.234235 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:52.234215 6234 scope.go:122] "RemoveContainer" containerID="65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361" Nov 03 22:47:52.234518 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:47:52.234488 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361\": container with ID starting with 65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361 not found: ID does not exist" containerID="65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361" Nov 03 22:47:52.234574 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:52.234528 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361"} err="failed to get container status \"65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361\": rpc error: code = NotFound desc = could not find container \"65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361\": container with ID starting with 65b39a9850546d5bda8abe24ed2ae094e4414abb71d6e409ef69fdba61d02361 not found: ID does not exist" Nov 03 22:47:52.377706 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:52.377670 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt"] Nov 03 22:47:52.425732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:52.425695 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mhbvt"] Nov 03 22:47:52.706038 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:52.706002 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" path="/var/lib/kubelet/pods/7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3/volumes" Nov 03 22:47:54.284662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:54.284619 6234 patch_prober.go:29] interesting pod/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:47:54.285067 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:54.284677 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:47:55.248765 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:55.248687 6234 generic.go:356] "Generic (PLEG): container finished" podID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerID="186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07" exitCode=0 Nov 03 22:47:55.248765 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:55.248748 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" event={"ID":"d8e787c8-9530-4516-ac62-2de3e352fdab","Type":"ContainerDied","Data":"186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07"} Nov 03 22:47:55.249282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:55.249262 6234 scope.go:122] "RemoveContainer" containerID="186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07" Nov 03 22:47:55.251338 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:55.251301 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:47:55.251436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:55.251404 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:47:55.251514 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:55.251459 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:47:56.288917 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:56.288877 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" event={"ID":"d8e787c8-9530-4516-ac62-2de3e352fdab","Type":"ContainerStarted","Data":"d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c"} Nov 03 22:47:56.960532 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:56.959931 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:47:56.964078 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:56.963746 6234 patch_prober.go:29] interesting pod/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:47:56.964078 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:56.963802 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 22:47:57.963521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:57.962628 6234 patch_prober.go:29] interesting pod/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:47:57.963521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:57.962691 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 22:47:58.947521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:58.946193 6234 patch_prober.go:29] interesting pod/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:47:58.947521 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:58.946253 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 22:47:58.998588 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:47:58.998549 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:47:59.964643 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:59.964607 6234 patch_prober.go:29] interesting pod/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:47:59.964825 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:47:59.964661 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 22:48:00.952520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:00.947287 6234 patch_prober.go:29] interesting pod/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:48:00.952520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:00.947433 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 22:48:01.947585 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:01.947553 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:48:01.948048 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:01.948028 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:48:01.948644 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:01.948625 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:48:12.453318 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:12.453275 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr"] Nov 03 22:48:12.453868 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:12.453588 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" containerID="cri-o://f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362" gracePeriod=30 Nov 03 22:48:17.713619 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.713560 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:48:17.816144 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816109 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-token\" (UniqueName: \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-token\") pod \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " Nov 03 22:48:17.816340 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816202 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-kube-api-access-s7wc7\" (UniqueName: \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-kube-api-access-s7wc7\") pod \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " Nov 03 22:48:17.816340 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816242 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-certs\") pod \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " Nov 03 22:48:17.816340 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816268 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-socket\") pod \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " Nov 03 22:48:17.816340 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816322 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-data\") pod \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " Nov 03 22:48:17.816611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816385 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istiod-ca-cert\") pod \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " Nov 03 22:48:17.816611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816480 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-credential-socket\") pod \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " Nov 03 22:48:17.816611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816575 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-podinfo\") pod \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " Nov 03 22:48:17.816611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816585 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-certs" pod "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" (UID: "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:17.816828 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816633 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-envoy\") pod \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\" (UID: \"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1\") " Nov 03 22:48:17.816828 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816639 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-socket" pod "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" (UID: "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:17.816828 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816696 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-data" pod "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" (UID: "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:17.816828 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816818 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-credential-socket" pod "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" (UID: "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:17.817046 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.816939 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istiod-ca-cert" pod "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" (UID: "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:17.817251 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.817227 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:17.817351 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.817254 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:17.817351 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.817271 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:17.817351 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.817283 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:17.817351 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.817297 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:17.818867 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.818841 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-token" pod "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" (UID: "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:17.819152 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.819115 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-podinfo" pod "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" (UID: "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:48:17.819152 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.819119 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-envoy" pod "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" (UID: "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:17.819453 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.819434 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-kube-api-access-s7wc7" pod "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" (UID: "1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1"). InnerVolumeSpecName "kube-api-access-s7wc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:17.919243 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.919204 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:17.919243 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.919244 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:17.919507 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.919260 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:17.919507 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:17.919274 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wc7\" (UniqueName: \"kubernetes.io/projected/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1-kube-api-access-s7wc7\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:18.424474 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:18.424436 6234 generic.go:356] "Generic (PLEG): container finished" podID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerID="f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362" exitCode=0 Nov 03 22:48:18.424681 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:18.424484 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" Nov 03 22:48:18.424681 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:18.424524 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" event={"ID":"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1","Type":"ContainerDied","Data":"f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362"} Nov 03 22:48:18.424681 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:18.424583 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr" event={"ID":"1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1","Type":"ContainerDied","Data":"89829ac5c5f11659b7709eebafb9817e35123cd9a4cf55c1444187101229a172"} Nov 03 22:48:18.424681 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:18.424609 6234 scope.go:122] "RemoveContainer" containerID="f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362" Nov 03 22:48:18.435547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:18.435526 6234 scope.go:122] "RemoveContainer" containerID="f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362" Nov 03 22:48:18.435796 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:48:18.435780 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362\": container with ID starting with f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362 not found: ID does not exist" containerID="f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362" Nov 03 22:48:18.435854 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:18.435804 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362"} err="failed to get container status \"f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362\": rpc error: code = NotFound desc = could not find container \"f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362\": container with ID starting with f8b08a8ed867aaa9e6c3d689a152caeb06b935b4af54818f12c571f4121cd362 not found: ID does not exist" Nov 03 22:48:18.613790 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:18.613757 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr"] Nov 03 22:48:18.660407 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:18.660375 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rvbgr"] Nov 03 22:48:18.706258 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:18.706229 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" path="/var/lib/kubelet/pods/1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1/volumes" Nov 03 22:48:20.603056 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:20.603025 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:48:28.996974 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:48:28.996937 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:48:58.985716 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:48:58.985680 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:48:58.997419 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:48:58.997386 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:49:28.996980 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:49:28.996944 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:49:53.049588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:49:53.049531 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:49:58.997092 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:49:58.997052 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:50:12.441025 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:12.440932 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg"] Nov 03 22:50:12.441635 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:12.441237 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" containerID="cri-o://eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4" gracePeriod=30 Nov 03 22:50:17.708891 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.708822 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:50:17.789117 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789077 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494f6493-e301-4f79-af6b-db4768de6a77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494f6493-e301-4f79-af6b-db4768de6a77-istiod-ca-cert\") pod \"494f6493-e301-4f79-af6b-db4768de6a77\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " Nov 03 22:50:17.789315 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789140 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-istio-token\" (UniqueName: \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-istio-token\") pod \"494f6493-e301-4f79-af6b-db4768de6a77\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " Nov 03 22:50:17.789315 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789180 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-kube-api-access-hrp2d\" (UniqueName: \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-kube-api-access-hrp2d\") pod \"494f6493-e301-4f79-af6b-db4768de6a77\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " Nov 03 22:50:17.789315 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789215 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-credential-socket\") pod \"494f6493-e301-4f79-af6b-db4768de6a77\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " Nov 03 22:50:17.789315 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789260 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-envoy\") pod \"494f6493-e301-4f79-af6b-db4768de6a77\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " Nov 03 22:50:17.789315 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789289 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-certs\") pod \"494f6493-e301-4f79-af6b-db4768de6a77\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " Nov 03 22:50:17.789618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789349 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-socket\") pod \"494f6493-e301-4f79-af6b-db4768de6a77\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " Nov 03 22:50:17.789618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789523 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/494f6493-e301-4f79-af6b-db4768de6a77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494f6493-e301-4f79-af6b-db4768de6a77-istio-podinfo\") pod \"494f6493-e301-4f79-af6b-db4768de6a77\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " Nov 03 22:50:17.789618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789597 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-data\") pod \"494f6493-e301-4f79-af6b-db4768de6a77\" (UID: \"494f6493-e301-4f79-af6b-db4768de6a77\") " Nov 03 22:50:17.789764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789596 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-certs" pod "494f6493-e301-4f79-af6b-db4768de6a77" (UID: "494f6493-e301-4f79-af6b-db4768de6a77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:17.789764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789629 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494f6493-e301-4f79-af6b-db4768de6a77-istiod-ca-cert" pod "494f6493-e301-4f79-af6b-db4768de6a77" (UID: "494f6493-e301-4f79-af6b-db4768de6a77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:17.789764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789667 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-socket" pod "494f6493-e301-4f79-af6b-db4768de6a77" (UID: "494f6493-e301-4f79-af6b-db4768de6a77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:17.789764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789732 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-credential-socket" pod "494f6493-e301-4f79-af6b-db4768de6a77" (UID: "494f6493-e301-4f79-af6b-db4768de6a77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:17.789951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.789890 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-data" pod "494f6493-e301-4f79-af6b-db4768de6a77" (UID: "494f6493-e301-4f79-af6b-db4768de6a77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:17.791058 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.790717 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:17.791058 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.790744 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494f6493-e301-4f79-af6b-db4768de6a77-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:17.791058 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.790759 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:17.791058 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.790773 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:17.791058 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.790788 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:17.791919 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.791892 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-envoy" pod "494f6493-e301-4f79-af6b-db4768de6a77" (UID: "494f6493-e301-4f79-af6b-db4768de6a77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:17.792130 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.792102 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-istio-token" pod "494f6493-e301-4f79-af6b-db4768de6a77" (UID: "494f6493-e301-4f79-af6b-db4768de6a77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:17.792130 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.792120 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-kube-api-access-hrp2d" pod "494f6493-e301-4f79-af6b-db4768de6a77" (UID: "494f6493-e301-4f79-af6b-db4768de6a77"). InnerVolumeSpecName "kube-api-access-hrp2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:17.792246 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.792155 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/494f6493-e301-4f79-af6b-db4768de6a77-istio-podinfo" pod "494f6493-e301-4f79-af6b-db4768de6a77" (UID: "494f6493-e301-4f79-af6b-db4768de6a77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:17.892433 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.892316 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:17.892433 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.892355 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrp2d\" (UniqueName: \"kubernetes.io/projected/494f6493-e301-4f79-af6b-db4768de6a77-kube-api-access-hrp2d\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:17.892433 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.892372 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494f6493-e301-4f79-af6b-db4768de6a77-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:17.892433 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:17.892386 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494f6493-e301-4f79-af6b-db4768de6a77-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:18.233218 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:18.233181 6234 generic.go:356] "Generic (PLEG): container finished" podID="494f6493-e301-4f79-af6b-db4768de6a77" containerID="eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4" exitCode=0 Nov 03 22:50:18.233395 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:18.233230 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" Nov 03 22:50:18.233395 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:18.233245 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" event={"ID":"494f6493-e301-4f79-af6b-db4768de6a77","Type":"ContainerDied","Data":"eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4"} Nov 03 22:50:18.233395 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:18.233299 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg" event={"ID":"494f6493-e301-4f79-af6b-db4768de6a77","Type":"ContainerDied","Data":"337cf9724918b25041e83db9408f7c7fc2bfe35474bd1ef7dfa6592bf22d3c12"} Nov 03 22:50:18.233395 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:18.233322 6234 scope.go:122] "RemoveContainer" containerID="eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4" Nov 03 22:50:18.249371 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:18.249346 6234 scope.go:122] "RemoveContainer" containerID="eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4" Nov 03 22:50:18.250364 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:50:18.250337 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4\": container with ID starting with eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4 not found: ID does not exist" containerID="eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4" Nov 03 22:50:18.250452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:18.250370 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4"} err="failed to get container status \"eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4\": rpc error: code = NotFound desc = could not find container \"eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4\": container with ID starting with eb592691d69cb6acff09728cfa5ac8e7831a030549e9f4e4891b5b435ccfa4b4 not found: ID does not exist" Nov 03 22:50:18.387042 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:18.387006 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg"] Nov 03 22:50:18.433371 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:18.433337 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zmphg"] Nov 03 22:50:18.705547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:18.705489 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494f6493-e301-4f79-af6b-db4768de6a77" path="/var/lib/kubelet/pods/494f6493-e301-4f79-af6b-db4768de6a77/volumes" Nov 03 22:50:19.697650 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:19.697614 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc"] Nov 03 22:50:19.744076 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:19.744043 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz"] Nov 03 22:50:28.996631 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:50:28.996593 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:50:34.719036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:34.718989 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" containerID="cri-o://d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361" gracePeriod=45 Nov 03 22:50:34.764167 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:34.764132 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" containerID="cri-o://a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b" gracePeriod=45 Nov 03 22:50:35.035875 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.035847 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:50:35.062892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.062869 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:50:35.114260 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114229 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-home\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-home\") pod \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " Nov 03 22:50:35.114431 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114367 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-dshm\") pod \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " Nov 03 22:50:35.114546 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114470 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a3ff55-de90-431d-b281-3f2333cb7f1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/71a3ff55-de90-431d-b281-3f2333cb7f1c-tls-certs\") pod \"71a3ff55-de90-431d-b281-3f2333cb7f1c\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " Nov 03 22:50:35.114614 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114538 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-home" pod "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" (UID: "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.114614 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114563 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-model-cache\") pod \"71a3ff55-de90-431d-b281-3f2333cb7f1c\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " Nov 03 22:50:35.114729 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114623 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tls-certs\") pod \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " Nov 03 22:50:35.114729 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114660 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-tmp-dir\") pod \"71a3ff55-de90-431d-b281-3f2333cb7f1c\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " Nov 03 22:50:35.114729 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114720 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-kube-api-access-2rr4s\" (UniqueName: \"kubernetes.io/projected/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-kube-api-access-2rr4s\") pod \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " Nov 03 22:50:35.114895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114757 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-model-cache\") pod \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " Nov 03 22:50:35.114895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114790 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a3ff55-de90-431d-b281-3f2333cb7f1c-kube-api-access-822lc\" (UniqueName: \"kubernetes.io/projected/71a3ff55-de90-431d-b281-3f2333cb7f1c-kube-api-access-822lc\") pod \"71a3ff55-de90-431d-b281-3f2333cb7f1c\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " Nov 03 22:50:35.114895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114823 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-dshm\") pod \"71a3ff55-de90-431d-b281-3f2333cb7f1c\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " Nov 03 22:50:35.114895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114852 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-model-cache" pod "71a3ff55-de90-431d-b281-3f2333cb7f1c" (UID: "71a3ff55-de90-431d-b281-3f2333cb7f1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.114895 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114880 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-home\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-home\") pod \"71a3ff55-de90-431d-b281-3f2333cb7f1c\" (UID: \"71a3ff55-de90-431d-b281-3f2333cb7f1c\") " Nov 03 22:50:35.115155 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.114950 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tmp-dir\") pod \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\" (UID: \"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2\") " Nov 03 22:50:35.116012 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.115765 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-model-cache" pod "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" (UID: "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.116400 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.116193 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-home" pod "71a3ff55-de90-431d-b281-3f2333cb7f1c" (UID: "71a3ff55-de90-431d-b281-3f2333cb7f1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.116607 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.116553 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-tmp-dir" pod "71a3ff55-de90-431d-b281-3f2333cb7f1c" (UID: "71a3ff55-de90-431d-b281-3f2333cb7f1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.116910 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.116882 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tmp-dir" pod "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" (UID: "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.117220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.117198 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.117323 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.117228 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.117323 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.117243 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.117323 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.117255 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.117323 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.117270 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.117323 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.117269 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-dshm" pod "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" (UID: "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.117323 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.117283 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.117924 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.117898 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-kube-api-access-2rr4s" pod "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" (UID: "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2"). InnerVolumeSpecName "kube-api-access-2rr4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:35.118289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.118234 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-dshm" pod "71a3ff55-de90-431d-b281-3f2333cb7f1c" (UID: "71a3ff55-de90-431d-b281-3f2333cb7f1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.118556 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.118520 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a3ff55-de90-431d-b281-3f2333cb7f1c-tls-certs" pod "71a3ff55-de90-431d-b281-3f2333cb7f1c" (UID: "71a3ff55-de90-431d-b281-3f2333cb7f1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:35.118975 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.118947 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a3ff55-de90-431d-b281-3f2333cb7f1c-kube-api-access-822lc" pod "71a3ff55-de90-431d-b281-3f2333cb7f1c" (UID: "71a3ff55-de90-431d-b281-3f2333cb7f1c"). InnerVolumeSpecName "kube-api-access-822lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:35.119056 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.118970 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tls-certs" pod "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" (UID: "2ecb248e-7c95-4a4d-9892-57d24a6b1dc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:35.218226 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.218197 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.218226 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.218225 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a3ff55-de90-431d-b281-3f2333cb7f1c-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.218226 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.218234 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.218464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.218242 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rr4s\" (UniqueName: \"kubernetes.io/projected/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2-kube-api-access-2rr4s\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.218464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.218251 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-822lc\" (UniqueName: \"kubernetes.io/projected/71a3ff55-de90-431d-b281-3f2333cb7f1c-kube-api-access-822lc\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.218464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.218260 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a3ff55-de90-431d-b281-3f2333cb7f1c-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.337799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.337711 6234 generic.go:356] "Generic (PLEG): container finished" podID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerID="a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b" exitCode=0 Nov 03 22:50:35.337799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.337757 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" Nov 03 22:50:35.338028 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.337796 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" event={"ID":"71a3ff55-de90-431d-b281-3f2333cb7f1c","Type":"ContainerDied","Data":"a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b"} Nov 03 22:50:35.338028 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.337846 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz" event={"ID":"71a3ff55-de90-431d-b281-3f2333cb7f1c","Type":"ContainerDied","Data":"a478dddb0e5efb00caae4fe4ecac13b399474596f1c001b91bbbcecf024ffc81"} Nov 03 22:50:35.338028 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.337871 6234 scope.go:122] "RemoveContainer" containerID="a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b" Nov 03 22:50:35.339436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.339328 6234 generic.go:356] "Generic (PLEG): container finished" podID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerID="d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361" exitCode=0 Nov 03 22:50:35.339436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.339367 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" event={"ID":"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2","Type":"ContainerDied","Data":"d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361"} Nov 03 22:50:35.339436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.339385 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" Nov 03 22:50:35.339436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.339401 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc" event={"ID":"2ecb248e-7c95-4a4d-9892-57d24a6b1dc2","Type":"ContainerDied","Data":"c5833aee302591dd73866f081659a90895ce0b5341b5eb40431be55fddf07d86"} Nov 03 22:50:35.348892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.348859 6234 scope.go:122] "RemoveContainer" containerID="a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b" Nov 03 22:50:35.349188 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:50:35.349166 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b\": container with ID starting with a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b not found: ID does not exist" containerID="a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b" Nov 03 22:50:35.349281 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.349193 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b"} err="failed to get container status \"a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b\": rpc error: code = NotFound desc = could not find container \"a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b\": container with ID starting with a8c2dafdc252242c706d76ab1080010a49dc298af96fb072e5bcca3cbae9848b not found: ID does not exist" Nov 03 22:50:35.349281 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.349210 6234 scope.go:122] "RemoveContainer" containerID="d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361" Nov 03 22:50:35.357853 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.357838 6234 scope.go:122] "RemoveContainer" containerID="d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361" Nov 03 22:50:35.358106 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:50:35.358088 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361\": container with ID starting with d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361 not found: ID does not exist" containerID="d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361" Nov 03 22:50:35.358165 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.358113 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361"} err="failed to get container status \"d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361\": rpc error: code = NotFound desc = could not find container \"d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361\": container with ID starting with d6454a19b80e8ab4d319758d984ac24d1c8c7aee0f1ce721cfd86a553f606361 not found: ID does not exist" Nov 03 22:50:35.490932 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.490899 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz"] Nov 03 22:50:35.494655 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.494628 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/simulator-w0-f61714-kserve-7c6b55bf5b-wgzjz"] Nov 03 22:50:35.595584 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.595481 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc"] Nov 03 22:50:35.599285 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:35.599262 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-2-w0-f61714-kserve-5775949fff-89qvc"] Nov 03 22:50:36.710143 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:36.710107 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" path="/var/lib/kubelet/pods/2ecb248e-7c95-4a4d-9892-57d24a6b1dc2/volumes" Nov 03 22:50:36.710790 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:36.710769 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" path="/var/lib/kubelet/pods/71a3ff55-de90-431d-b281-3f2333cb7f1c/volumes" Nov 03 22:50:58.996452 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:50:58.996406 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:50:59.635359 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:50:59.635316 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:51:07.304271 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:07.304226 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv"] Nov 03 22:51:07.304791 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:07.304447 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" containerID="cri-o://36c3dea3aa5eea7967ef8302e7a6a14265e7cf55aa6d512b7f4d3bc6add320f1" gracePeriod=30 Nov 03 22:51:07.435824 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:07.435787 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p"] Nov 03 22:51:07.547894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:07.547856 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw"] Nov 03 22:51:12.436146 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.436068 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" podUID="a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f" containerName="payload-processing" containerID="cri-o://8692bae8a9516d341b8aef969fed39a98aa76ccd84571c715b82522df8807e65" gracePeriod=25 Nov 03 22:51:12.548330 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.548279 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" podUID="fd9fcdcb-09a9-4450-91a4-2c243670093a" containerName="payload-pre-processing" containerID="cri-o://dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702" gracePeriod=25 Nov 03 22:51:12.616779 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.616731 6234 generic.go:356] "Generic (PLEG): container finished" podID="a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f" containerID="8692bae8a9516d341b8aef969fed39a98aa76ccd84571c715b82522df8807e65" exitCode=0 Nov 03 22:51:12.617196 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.617136 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" event={"ID":"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f","Type":"ContainerDied","Data":"8692bae8a9516d341b8aef969fed39a98aa76ccd84571c715b82522df8807e65"} Nov 03 22:51:12.619363 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.619198 6234 generic.go:356] "Generic (PLEG): container finished" podID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerID="36c3dea3aa5eea7967ef8302e7a6a14265e7cf55aa6d512b7f4d3bc6add320f1" exitCode=0 Nov 03 22:51:12.619363 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.619264 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" event={"ID":"a0fa3225-0006-485d-a7de-59dd408b6f99","Type":"ContainerDied","Data":"36c3dea3aa5eea7967ef8302e7a6a14265e7cf55aa6d512b7f4d3bc6add320f1"} Nov 03 22:51:12.619363 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.619311 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" event={"ID":"a0fa3225-0006-485d-a7de-59dd408b6f99","Type":"ContainerDied","Data":"31e89243dd18794f6b568fdb9bcbe572259f9a14ba72c1c4d78537b785ef938b"} Nov 03 22:51:12.619363 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.619336 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31e89243dd18794f6b568fdb9bcbe572259f9a14ba72c1c4d78537b785ef938b" Nov 03 22:51:12.652732 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.652701 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:51:12.690474 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.690448 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:51:12.804592 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.804567 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:51:12.827063 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.826979 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-certs\") pod \"a0fa3225-0006-485d-a7de-59dd408b6f99\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " Nov 03 22:51:12.827063 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827029 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-credential-socket\") pod \"a0fa3225-0006-485d-a7de-59dd408b6f99\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " Nov 03 22:51:12.827280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827096 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-socket\") pod \"a0fa3225-0006-485d-a7de-59dd408b6f99\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " Nov 03 22:51:12.827280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827137 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-kube-api-access-l2gcr\" (UniqueName: \"kubernetes.io/projected/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-kube-api-access-l2gcr\") pod \"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f\" (UID: \"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f\") " Nov 03 22:51:12.827280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827191 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-data\") pod \"a0fa3225-0006-485d-a7de-59dd408b6f99\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " Nov 03 22:51:12.827280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827261 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-podinfo\") pod \"a0fa3225-0006-485d-a7de-59dd408b6f99\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " Nov 03 22:51:12.827544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827294 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-certs" pod "a0fa3225-0006-485d-a7de-59dd408b6f99" (UID: "a0fa3225-0006-485d-a7de-59dd408b6f99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:12.827544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827310 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-token\" (UniqueName: \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-token\") pod \"a0fa3225-0006-485d-a7de-59dd408b6f99\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " Nov 03 22:51:12.827544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827400 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-kube-api-access-l994b\" (UniqueName: \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-kube-api-access-l994b\") pod \"a0fa3225-0006-485d-a7de-59dd408b6f99\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " Nov 03 22:51:12.827544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827423 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-socket" pod "a0fa3225-0006-485d-a7de-59dd408b6f99" (UID: "a0fa3225-0006-485d-a7de-59dd408b6f99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:12.827544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827463 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fa3225-0006-485d-a7de-59dd408b6f99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fa3225-0006-485d-a7de-59dd408b6f99-istiod-ca-cert\") pod \"a0fa3225-0006-485d-a7de-59dd408b6f99\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " Nov 03 22:51:12.827544 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827511 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-plugins-config-volume\") pod \"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f\" (UID: \"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f\") " Nov 03 22:51:12.827886 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827548 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-envoy\") pod \"a0fa3225-0006-485d-a7de-59dd408b6f99\" (UID: \"a0fa3225-0006-485d-a7de-59dd408b6f99\") " Nov 03 22:51:12.827886 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827763 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-data" pod "a0fa3225-0006-485d-a7de-59dd408b6f99" (UID: "a0fa3225-0006-485d-a7de-59dd408b6f99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:12.828002 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.827980 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-credential-socket" pod "a0fa3225-0006-485d-a7de-59dd408b6f99" (UID: "a0fa3225-0006-485d-a7de-59dd408b6f99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:12.828397 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.828373 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.828524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.828402 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.828524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.828405 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-plugins-config-volume" pod "a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f" (UID: "a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:12.828524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.828418 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.828524 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.828433 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.829875 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.829630 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fa3225-0006-485d-a7de-59dd408b6f99-istiod-ca-cert" pod "a0fa3225-0006-485d-a7de-59dd408b6f99" (UID: "a0fa3225-0006-485d-a7de-59dd408b6f99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:12.830326 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.830301 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-token" pod "a0fa3225-0006-485d-a7de-59dd408b6f99" (UID: "a0fa3225-0006-485d-a7de-59dd408b6f99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:12.830887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.830860 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-kube-api-access-l2gcr" pod "a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f" (UID: "a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f"). InnerVolumeSpecName "kube-api-access-l2gcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:12.831350 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.831326 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-podinfo" pod "a0fa3225-0006-485d-a7de-59dd408b6f99" (UID: "a0fa3225-0006-485d-a7de-59dd408b6f99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:12.831540 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.831513 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-envoy" pod "a0fa3225-0006-485d-a7de-59dd408b6f99" (UID: "a0fa3225-0006-485d-a7de-59dd408b6f99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:12.832060 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.832044 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-kube-api-access-l994b" pod "a0fa3225-0006-485d-a7de-59dd408b6f99" (UID: "a0fa3225-0006-485d-a7de-59dd408b6f99"). InnerVolumeSpecName "kube-api-access-l994b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:12.928915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.928884 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9fcdcb-09a9-4450-91a4-2c243670093a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9fcdcb-09a9-4450-91a4-2c243670093a-plugins-config-volume\") pod \"fd9fcdcb-09a9-4450-91a4-2c243670093a\" (UID: \"fd9fcdcb-09a9-4450-91a4-2c243670093a\") " Nov 03 22:51:12.929084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.928981 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9fcdcb-09a9-4450-91a4-2c243670093a-kube-api-access-tpgw2\" (UniqueName: \"kubernetes.io/projected/fd9fcdcb-09a9-4450-91a4-2c243670093a-kube-api-access-tpgw2\") pod \"fd9fcdcb-09a9-4450-91a4-2c243670093a\" (UID: \"fd9fcdcb-09a9-4450-91a4-2c243670093a\") " Nov 03 22:51:12.929294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.929265 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9fcdcb-09a9-4450-91a4-2c243670093a-plugins-config-volume" pod "fd9fcdcb-09a9-4450-91a4-2c243670093a" (UID: "fd9fcdcb-09a9-4450-91a4-2c243670093a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:12.929387 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.929370 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9fcdcb-09a9-4450-91a4-2c243670093a-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.929430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.929393 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2gcr\" (UniqueName: \"kubernetes.io/projected/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-kube-api-access-l2gcr\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.929430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.929404 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.929430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.929417 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.929573 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.929432 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l994b\" (UniqueName: \"kubernetes.io/projected/a0fa3225-0006-485d-a7de-59dd408b6f99-kube-api-access-l994b\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.929573 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.929447 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fa3225-0006-485d-a7de-59dd408b6f99-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.929573 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.929460 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.929573 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.929473 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fa3225-0006-485d-a7de-59dd408b6f99-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:12.931072 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:12.931050 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9fcdcb-09a9-4450-91a4-2c243670093a-kube-api-access-tpgw2" pod "fd9fcdcb-09a9-4450-91a4-2c243670093a" (UID: "fd9fcdcb-09a9-4450-91a4-2c243670093a"). InnerVolumeSpecName "kube-api-access-tpgw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.033955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.031748 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpgw2\" (UniqueName: \"kubernetes.io/projected/fd9fcdcb-09a9-4450-91a4-2c243670093a-kube-api-access-tpgw2\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.628894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.628815 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" Nov 03 22:51:13.628894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.628829 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p" event={"ID":"a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f","Type":"ContainerDied","Data":"1f0f970d112c23257ccad982e79886c7eaaddcf49930a01985f013a6ccfaf634"} Nov 03 22:51:13.628894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.628890 6234 scope.go:122] "RemoveContainer" containerID="8692bae8a9516d341b8aef969fed39a98aa76ccd84571c715b82522df8807e65" Nov 03 22:51:13.630297 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.630237 6234 generic.go:356] "Generic (PLEG): container finished" podID="fd9fcdcb-09a9-4450-91a4-2c243670093a" containerID="dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702" exitCode=0 Nov 03 22:51:13.630297 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.630287 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" Nov 03 22:51:13.630409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.630323 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" event={"ID":"fd9fcdcb-09a9-4450-91a4-2c243670093a","Type":"ContainerDied","Data":"dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702"} Nov 03 22:51:13.630409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.630382 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw" event={"ID":"fd9fcdcb-09a9-4450-91a4-2c243670093a","Type":"ContainerDied","Data":"06cd31f83f1dbc14a90129105aeb8e8fbc159d53f2dae83a27d597cd9aa256de"} Nov 03 22:51:13.630489 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.630449 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv" Nov 03 22:51:13.640899 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.640863 6234 scope.go:122] "RemoveContainer" containerID="dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702" Nov 03 22:51:13.649660 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.649639 6234 scope.go:122] "RemoveContainer" containerID="dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702" Nov 03 22:51:13.649925 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:51:13.649907 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702\": container with ID starting with dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702 not found: ID does not exist" containerID="dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702" Nov 03 22:51:13.649973 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.649935 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702"} err="failed to get container status \"dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702\": rpc error: code = NotFound desc = could not find container \"dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702\": container with ID starting with dd625260303e87febece730ededd23f454077091af72b21dc36f1d0749ead702 not found: ID does not exist" Nov 03 22:51:13.783240 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.783210 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv"] Nov 03 22:51:13.786800 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.786773 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-j5xlv"] Nov 03 22:51:13.891211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.891132 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw"] Nov 03 22:51:13.896719 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.896691 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-295682-7599fdc787-4kfmw"] Nov 03 22:51:13.999008 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:13.998979 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p"] Nov 03 22:51:14.002429 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:14.002400 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-295682-7d66f469b-bdf7p"] Nov 03 22:51:14.712481 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:14.712452 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" path="/var/lib/kubelet/pods/a0fa3225-0006-485d-a7de-59dd408b6f99/volumes" Nov 03 22:51:14.712971 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:14.712955 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f" path="/var/lib/kubelet/pods/a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f/volumes" Nov 03 22:51:14.713364 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:14.713351 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9fcdcb-09a9-4450-91a4-2c243670093a" path="/var/lib/kubelet/pods/fd9fcdcb-09a9-4450-91a4-2c243670093a/volumes" Nov 03 22:51:27.012771 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:27.012736 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw"] Nov 03 22:51:27.013189 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:27.012973 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" containerID="cri-o://a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db" gracePeriod=30 Nov 03 22:51:28.996292 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:51:28.996249 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:51:32.301437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.301412 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:51:32.416254 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416221 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-credential-socket\") pod \"c6ab3852-4c26-42b9-9927-926bb9a00e79\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " Nov 03 22:51:32.416410 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416289 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-envoy\") pod \"c6ab3852-4c26-42b9-9927-926bb9a00e79\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " Nov 03 22:51:32.416410 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416349 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-kube-api-access-2r94n\" (UniqueName: \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-kube-api-access-2r94n\") pod \"c6ab3852-4c26-42b9-9927-926bb9a00e79\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " Nov 03 22:51:32.416410 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416404 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-data\") pod \"c6ab3852-4c26-42b9-9927-926bb9a00e79\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " Nov 03 22:51:32.416610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416457 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ab3852-4c26-42b9-9927-926bb9a00e79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ab3852-4c26-42b9-9927-926bb9a00e79-istiod-ca-cert\") pod \"c6ab3852-4c26-42b9-9927-926bb9a00e79\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " Nov 03 22:51:32.416610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416481 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-credential-socket" pod "c6ab3852-4c26-42b9-9927-926bb9a00e79" (UID: "c6ab3852-4c26-42b9-9927-926bb9a00e79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:32.416610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416490 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-certs\") pod \"c6ab3852-4c26-42b9-9927-926bb9a00e79\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " Nov 03 22:51:32.416610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416556 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-socket\") pod \"c6ab3852-4c26-42b9-9927-926bb9a00e79\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " Nov 03 22:51:32.416610 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416589 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-token\" (UniqueName: \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-token\") pod \"c6ab3852-4c26-42b9-9927-926bb9a00e79\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " Nov 03 22:51:32.416935 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416664 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-podinfo\") pod \"c6ab3852-4c26-42b9-9927-926bb9a00e79\" (UID: \"c6ab3852-4c26-42b9-9927-926bb9a00e79\") " Nov 03 22:51:32.416935 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416745 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-data" pod "c6ab3852-4c26-42b9-9927-926bb9a00e79" (UID: "c6ab3852-4c26-42b9-9927-926bb9a00e79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:32.416935 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416803 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-certs" pod "c6ab3852-4c26-42b9-9927-926bb9a00e79" (UID: "c6ab3852-4c26-42b9-9927-926bb9a00e79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:32.416935 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416837 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-socket" pod "c6ab3852-4c26-42b9-9927-926bb9a00e79" (UID: "c6ab3852-4c26-42b9-9927-926bb9a00e79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:32.416935 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.416889 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ab3852-4c26-42b9-9927-926bb9a00e79-istiod-ca-cert" pod "c6ab3852-4c26-42b9-9927-926bb9a00e79" (UID: "c6ab3852-4c26-42b9-9927-926bb9a00e79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:32.417255 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.417229 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:32.417373 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.417258 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:32.417373 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.417272 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ab3852-4c26-42b9-9927-926bb9a00e79-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:32.417373 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.417287 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:32.417373 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.417301 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:32.418490 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.418467 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-envoy" pod "c6ab3852-4c26-42b9-9927-926bb9a00e79" (UID: "c6ab3852-4c26-42b9-9927-926bb9a00e79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:32.418898 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.418876 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-token" pod "c6ab3852-4c26-42b9-9927-926bb9a00e79" (UID: "c6ab3852-4c26-42b9-9927-926bb9a00e79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:32.418979 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.418927 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-podinfo" pod "c6ab3852-4c26-42b9-9927-926bb9a00e79" (UID: "c6ab3852-4c26-42b9-9927-926bb9a00e79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:32.419026 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.418996 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-kube-api-access-2r94n" pod "c6ab3852-4c26-42b9-9927-926bb9a00e79" (UID: "c6ab3852-4c26-42b9-9927-926bb9a00e79"). InnerVolumeSpecName "kube-api-access-2r94n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:32.518547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.518486 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:32.518547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.518550 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r94n\" (UniqueName: \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-kube-api-access-2r94n\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:32.518746 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.518563 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:32.518746 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.518571 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ab3852-4c26-42b9-9927-926bb9a00e79-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:32.750798 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.750766 6234 generic.go:356] "Generic (PLEG): container finished" podID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerID="a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db" exitCode=0 Nov 03 22:51:32.750956 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.750816 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" Nov 03 22:51:32.750956 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.750842 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" event={"ID":"c6ab3852-4c26-42b9-9927-926bb9a00e79","Type":"ContainerDied","Data":"a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db"} Nov 03 22:51:32.750956 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.750884 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw" event={"ID":"c6ab3852-4c26-42b9-9927-926bb9a00e79","Type":"ContainerDied","Data":"77672c4c7c255291bf56c9b73ed646eed6d579b555f99ebc564b94e4e9ef339e"} Nov 03 22:51:32.750956 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.750906 6234 scope.go:122] "RemoveContainer" containerID="a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db" Nov 03 22:51:32.760428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.760403 6234 scope.go:122] "RemoveContainer" containerID="19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea" Nov 03 22:51:32.769411 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.769390 6234 scope.go:122] "RemoveContainer" containerID="a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db" Nov 03 22:51:32.769773 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:51:32.769752 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db\": container with ID starting with a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db not found: ID does not exist" containerID="a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db" Nov 03 22:51:32.769842 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.769784 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db"} err="failed to get container status \"a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db\": rpc error: code = NotFound desc = could not find container \"a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db\": container with ID starting with a4e9b6dfb882b9f05f4b47b97a328522f165ddedd4db297d4bfd197599f3f7db not found: ID does not exist" Nov 03 22:51:32.769842 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.769802 6234 scope.go:122] "RemoveContainer" containerID="19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea" Nov 03 22:51:32.770042 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:51:32.770028 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea\": container with ID starting with 19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea not found: ID does not exist" containerID="19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea" Nov 03 22:51:32.770082 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.770046 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea"} err="failed to get container status \"19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea\": rpc error: code = NotFound desc = could not find container \"19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea\": container with ID starting with 19b1a3b9a045e9992cbaa1cc060ec0d40305403478abd99f615c2c1cfd1955ea not found: ID does not exist" Nov 03 22:51:32.905464 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.905423 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw"] Nov 03 22:51:32.952203 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:32.952174 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-345da9-openshift-default-6fd8dc4d8c-4x6mw"] Nov 03 22:51:34.704339 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:34.704307 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" path="/var/lib/kubelet/pods/c6ab3852-4c26-42b9-9927-926bb9a00e79/volumes" Nov 03 22:51:54.072894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:54.072862 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr"] Nov 03 22:51:54.119616 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:51:54.119583 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph"] Nov 03 22:51:58.997110 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:51:58.997068 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:52:01.820130 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:01.820095 6234 scope.go:122] "RemoveContainer" containerID="36c3dea3aa5eea7967ef8302e7a6a14265e7cf55aa6d512b7f4d3bc6add320f1" Nov 03 22:52:03.186267 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:03.186232 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:52:06.751819 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:06.751783 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8"] Nov 03 22:52:06.752210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:06.752016 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" containerID="cri-o://bc8f2571f3586cfe9cf1ce7f9400e3cedf9bf6127b6d372f1f36e05463acb24f" gracePeriod=30 Nov 03 22:52:08.047834 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:08.047797 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn"] Nov 03 22:52:08.048342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:08.048007 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" containerID="cri-o://d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c" gracePeriod=30 Nov 03 22:52:09.093969 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.093922 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" containerID="cri-o://b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0" gracePeriod=45 Nov 03 22:52:09.140206 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.140158 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" containerID="cri-o://05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2" gracePeriod=45 Nov 03 22:52:09.397656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.397629 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:52:09.430436 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.430408 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:52:09.484531 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.484475 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-home\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-home\") pod \"4ca0821f-278d-456f-82e9-31578aa0b3a6\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " Nov 03 22:52:09.484701 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.484633 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-tmp-dir\") pod \"4ca0821f-278d-456f-82e9-31578aa0b3a6\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " Nov 03 22:52:09.484701 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.484672 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-dshm\") pod \"4ca0821f-278d-456f-82e9-31578aa0b3a6\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " Nov 03 22:52:09.484773 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.484751 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca0821f-278d-456f-82e9-31578aa0b3a6-kube-api-access-vx422\" (UniqueName: \"kubernetes.io/projected/4ca0821f-278d-456f-82e9-31578aa0b3a6-kube-api-access-vx422\") pod \"4ca0821f-278d-456f-82e9-31578aa0b3a6\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " Nov 03 22:52:09.484819 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.484776 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-home" pod "4ca0821f-278d-456f-82e9-31578aa0b3a6" (UID: "4ca0821f-278d-456f-82e9-31578aa0b3a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.484819 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.484797 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ca0821f-278d-456f-82e9-31578aa0b3a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca0821f-278d-456f-82e9-31578aa0b3a6-tls-certs\") pod \"4ca0821f-278d-456f-82e9-31578aa0b3a6\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " Nov 03 22:52:09.484901 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.484832 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-model-cache\") pod \"4ca0821f-278d-456f-82e9-31578aa0b3a6\" (UID: \"4ca0821f-278d-456f-82e9-31578aa0b3a6\") " Nov 03 22:52:09.484939 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.484896 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-tmp-dir" pod "4ca0821f-278d-456f-82e9-31578aa0b3a6" (UID: "4ca0821f-278d-456f-82e9-31578aa0b3a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.485176 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.485147 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-model-cache" pod "4ca0821f-278d-456f-82e9-31578aa0b3a6" (UID: "4ca0821f-278d-456f-82e9-31578aa0b3a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.485353 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.485336 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.485409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.485358 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.485409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.485372 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.487001 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.486981 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca0821f-278d-456f-82e9-31578aa0b3a6-tls-certs" pod "4ca0821f-278d-456f-82e9-31578aa0b3a6" (UID: "4ca0821f-278d-456f-82e9-31578aa0b3a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:09.487236 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.487211 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-dshm" pod "4ca0821f-278d-456f-82e9-31578aa0b3a6" (UID: "4ca0821f-278d-456f-82e9-31578aa0b3a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.487324 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.487231 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca0821f-278d-456f-82e9-31578aa0b3a6-kube-api-access-vx422" pod "4ca0821f-278d-456f-82e9-31578aa0b3a6" (UID: "4ca0821f-278d-456f-82e9-31578aa0b3a6"). InnerVolumeSpecName "kube-api-access-vx422". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:09.586488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.586456 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6866d963-bd92-4452-be98-9bdd6a7e9e11-kube-api-access-hsnht\" (UniqueName: \"kubernetes.io/projected/6866d963-bd92-4452-be98-9bdd6a7e9e11-kube-api-access-hsnht\") pod \"6866d963-bd92-4452-be98-9bdd6a7e9e11\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " Nov 03 22:52:09.586684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.586507 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6866d963-bd92-4452-be98-9bdd6a7e9e11-tls-certs\" (UniqueName: \"kubernetes.io/secret/6866d963-bd92-4452-be98-9bdd6a7e9e11-tls-certs\") pod \"6866d963-bd92-4452-be98-9bdd6a7e9e11\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " Nov 03 22:52:09.586684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.586583 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-tmp-dir\") pod \"6866d963-bd92-4452-be98-9bdd6a7e9e11\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " Nov 03 22:52:09.586684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.586630 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-dshm\") pod \"6866d963-bd92-4452-be98-9bdd6a7e9e11\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " Nov 03 22:52:09.586861 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.586697 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-model-cache\") pod \"6866d963-bd92-4452-be98-9bdd6a7e9e11\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " Nov 03 22:52:09.586861 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.586728 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-home\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-home\") pod \"6866d963-bd92-4452-be98-9bdd6a7e9e11\" (UID: \"6866d963-bd92-4452-be98-9bdd6a7e9e11\") " Nov 03 22:52:09.587053 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.586897 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-tmp-dir" pod "6866d963-bd92-4452-be98-9bdd6a7e9e11" (UID: "6866d963-bd92-4452-be98-9bdd6a7e9e11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.587053 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.586976 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-model-cache" pod "6866d963-bd92-4452-be98-9bdd6a7e9e11" (UID: "6866d963-bd92-4452-be98-9bdd6a7e9e11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.587169 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.587110 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-home" pod "6866d963-bd92-4452-be98-9bdd6a7e9e11" (UID: "6866d963-bd92-4452-be98-9bdd6a7e9e11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.587279 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.587258 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca0821f-278d-456f-82e9-31578aa0b3a6-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.587338 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.587284 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.587338 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.587301 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx422\" (UniqueName: \"kubernetes.io/projected/4ca0821f-278d-456f-82e9-31578aa0b3a6-kube-api-access-vx422\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.587338 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.587318 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca0821f-278d-456f-82e9-31578aa0b3a6-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.587338 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.587331 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.587563 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.587345 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.588706 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.588680 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-dshm" pod "6866d963-bd92-4452-be98-9bdd6a7e9e11" (UID: "6866d963-bd92-4452-be98-9bdd6a7e9e11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.588832 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.588767 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6866d963-bd92-4452-be98-9bdd6a7e9e11-tls-certs" pod "6866d963-bd92-4452-be98-9bdd6a7e9e11" (UID: "6866d963-bd92-4452-be98-9bdd6a7e9e11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:09.588913 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.588890 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6866d963-bd92-4452-be98-9bdd6a7e9e11-kube-api-access-hsnht" pod "6866d963-bd92-4452-be98-9bdd6a7e9e11" (UID: "6866d963-bd92-4452-be98-9bdd6a7e9e11"). InnerVolumeSpecName "kube-api-access-hsnht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:09.688622 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.688594 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6866d963-bd92-4452-be98-9bdd6a7e9e11-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.688754 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.688647 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsnht\" (UniqueName: \"kubernetes.io/projected/6866d963-bd92-4452-be98-9bdd6a7e9e11-kube-api-access-hsnht\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.688754 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.688659 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6866d963-bd92-4452-be98-9bdd6a7e9e11-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.978899 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.978798 6234 generic.go:356] "Generic (PLEG): container finished" podID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerID="05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2" exitCode=0 Nov 03 22:52:09.978899 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.978869 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" Nov 03 22:52:09.978899 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.978878 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" event={"ID":"6866d963-bd92-4452-be98-9bdd6a7e9e11","Type":"ContainerDied","Data":"05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2"} Nov 03 22:52:09.979180 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.978919 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph" event={"ID":"6866d963-bd92-4452-be98-9bdd6a7e9e11","Type":"ContainerDied","Data":"4d104854f7c24febcefd45c33b3ef28a1fe95cc66d4bb506b3422d7036f4e8df"} Nov 03 22:52:09.979180 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.978933 6234 scope.go:122] "RemoveContainer" containerID="05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2" Nov 03 22:52:09.980460 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.980425 6234 generic.go:356] "Generic (PLEG): container finished" podID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerID="b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0" exitCode=0 Nov 03 22:52:09.980625 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.980525 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" Nov 03 22:52:09.980625 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.980543 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" event={"ID":"4ca0821f-278d-456f-82e9-31578aa0b3a6","Type":"ContainerDied","Data":"b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0"} Nov 03 22:52:09.980625 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.980579 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr" event={"ID":"4ca0821f-278d-456f-82e9-31578aa0b3a6","Type":"ContainerDied","Data":"8adf3e7530bd2a2fbfd187f8517142eb0602f6ae3d26062050d043ca1c80bc7a"} Nov 03 22:52:09.989959 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.989942 6234 scope.go:122] "RemoveContainer" containerID="05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2" Nov 03 22:52:09.990227 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:09.990205 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2\": container with ID starting with 05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2 not found: ID does not exist" containerID="05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2" Nov 03 22:52:09.990303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.990236 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2"} err="failed to get container status \"05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2\": rpc error: code = NotFound desc = could not find container \"05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2\": container with ID starting with 05100d779a31dc2d2ebab1ee72c44e0f6cf53b182aebf32303f4b980f7195ad2 not found: ID does not exist" Nov 03 22:52:09.990303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:09.990252 6234 scope.go:122] "RemoveContainer" containerID="b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0" Nov 03 22:52:10.000370 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:10.000346 6234 scope.go:122] "RemoveContainer" containerID="b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0" Nov 03 22:52:10.002667 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:10.002605 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0\": container with ID starting with b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0 not found: ID does not exist" containerID="b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0" Nov 03 22:52:10.002667 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:10.002639 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0"} err="failed to get container status \"b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0\": rpc error: code = NotFound desc = could not find container \"b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0\": container with ID starting with b5ff0467229d6f58b24d37eab2d5d2f0a15b6c86f0bf2e7002839203469565e0 not found: ID does not exist" Nov 03 22:52:10.133184 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:10.133153 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph"] Nov 03 22:52:10.179664 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:10.179636 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/premium-w3-41bb2b-kserve-779f8cfc8b-rpmph"] Nov 03 22:52:10.281688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:10.281609 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr"] Nov 03 22:52:10.285357 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:10.285329 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-w3-41bb2b-kserve-658f8c5cd5-x8hdr"] Nov 03 22:52:10.704563 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:10.704524 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" path="/var/lib/kubelet/pods/4ca0821f-278d-456f-82e9-31578aa0b3a6/volumes" Nov 03 22:52:10.705048 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:10.705026 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" path="/var/lib/kubelet/pods/6866d963-bd92-4452-be98-9bdd6a7e9e11/volumes" Nov 03 22:52:11.995467 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:11.995423 6234 generic.go:356] "Generic (PLEG): container finished" podID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerID="bc8f2571f3586cfe9cf1ce7f9400e3cedf9bf6127b6d372f1f36e05463acb24f" exitCode=0 Nov 03 22:52:11.995877 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:11.995529 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" event={"ID":"57101afc-e8db-439a-b8fe-ba005d025ea3","Type":"ContainerDied","Data":"bc8f2571f3586cfe9cf1ce7f9400e3cedf9bf6127b6d372f1f36e05463acb24f"} Nov 03 22:52:12.019020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.018961 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:52:12.117845 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.117811 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57101afc-e8db-439a-b8fe-ba005d025ea3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57101afc-e8db-439a-b8fe-ba005d025ea3-istiod-ca-cert\") pod \"57101afc-e8db-439a-b8fe-ba005d025ea3\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " Nov 03 22:52:12.118067 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.117959 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-kube-api-access-5zvz6\" (UniqueName: \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-kube-api-access-5zvz6\") pod \"57101afc-e8db-439a-b8fe-ba005d025ea3\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " Nov 03 22:52:12.118067 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118022 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-token\" (UniqueName: \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-token\") pod \"57101afc-e8db-439a-b8fe-ba005d025ea3\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " Nov 03 22:52:12.118067 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118062 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-data\") pod \"57101afc-e8db-439a-b8fe-ba005d025ea3\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " Nov 03 22:52:12.118247 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118119 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-certs\") pod \"57101afc-e8db-439a-b8fe-ba005d025ea3\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " Nov 03 22:52:12.118247 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118207 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-credential-socket\") pod \"57101afc-e8db-439a-b8fe-ba005d025ea3\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " Nov 03 22:52:12.118247 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118233 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57101afc-e8db-439a-b8fe-ba005d025ea3-istiod-ca-cert" pod "57101afc-e8db-439a-b8fe-ba005d025ea3" (UID: "57101afc-e8db-439a-b8fe-ba005d025ea3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:12.118402 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118266 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-envoy\") pod \"57101afc-e8db-439a-b8fe-ba005d025ea3\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " Nov 03 22:52:12.118402 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118305 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-podinfo\") pod \"57101afc-e8db-439a-b8fe-ba005d025ea3\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " Nov 03 22:52:12.118557 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118516 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-data" pod "57101afc-e8db-439a-b8fe-ba005d025ea3" (UID: "57101afc-e8db-439a-b8fe-ba005d025ea3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:12.118634 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118565 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-credential-socket" pod "57101afc-e8db-439a-b8fe-ba005d025ea3" (UID: "57101afc-e8db-439a-b8fe-ba005d025ea3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:12.118792 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118771 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-certs" pod "57101afc-e8db-439a-b8fe-ba005d025ea3" (UID: "57101afc-e8db-439a-b8fe-ba005d025ea3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:12.118870 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.118800 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-socket" pod "57101afc-e8db-439a-b8fe-ba005d025ea3" (UID: "57101afc-e8db-439a-b8fe-ba005d025ea3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:12.119065 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.119035 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-socket\") pod \"57101afc-e8db-439a-b8fe-ba005d025ea3\" (UID: \"57101afc-e8db-439a-b8fe-ba005d025ea3\") " Nov 03 22:52:12.119665 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.119641 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.119781 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.119671 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57101afc-e8db-439a-b8fe-ba005d025ea3-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.119781 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.119688 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.119781 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.119706 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.119781 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.119721 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.120797 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.120768 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-envoy" pod "57101afc-e8db-439a-b8fe-ba005d025ea3" (UID: "57101afc-e8db-439a-b8fe-ba005d025ea3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:12.120912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.120774 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-token" pod "57101afc-e8db-439a-b8fe-ba005d025ea3" (UID: "57101afc-e8db-439a-b8fe-ba005d025ea3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:12.120912 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.120815 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-kube-api-access-5zvz6" pod "57101afc-e8db-439a-b8fe-ba005d025ea3" (UID: "57101afc-e8db-439a-b8fe-ba005d025ea3"). InnerVolumeSpecName "kube-api-access-5zvz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:12.121137 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.121116 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-podinfo" pod "57101afc-e8db-439a-b8fe-ba005d025ea3" (UID: "57101afc-e8db-439a-b8fe-ba005d025ea3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:12.221201 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.221167 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zvz6\" (UniqueName: \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-kube-api-access-5zvz6\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.221201 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.221194 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.221201 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.221204 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.221201 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:12.221212 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57101afc-e8db-439a-b8fe-ba005d025ea3-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:13.002547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.002443 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" event={"ID":"57101afc-e8db-439a-b8fe-ba005d025ea3","Type":"ContainerDied","Data":"b94197d6032913fd9b4be9d4ef4d796cc95a764e99f15dadcac02c28adc832c9"} Nov 03 22:52:13.002547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.002451 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8" Nov 03 22:52:13.002547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.002528 6234 scope.go:122] "RemoveContainer" containerID="bc8f2571f3586cfe9cf1ce7f9400e3cedf9bf6127b6d372f1f36e05463acb24f" Nov 03 22:52:13.203126 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.203094 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8"] Nov 03 22:52:13.264894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.264791 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-8rnn8"] Nov 03 22:52:13.327557 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.327535 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:52:13.437410 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437384 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-socket\") pod \"d8e787c8-9530-4516-ac62-2de3e352fdab\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " Nov 03 22:52:13.437587 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437427 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-podinfo\") pod \"d8e787c8-9530-4516-ac62-2de3e352fdab\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " Nov 03 22:52:13.437587 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437451 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-kube-api-access-bjx79\" (UniqueName: \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-kube-api-access-bjx79\") pod \"d8e787c8-9530-4516-ac62-2de3e352fdab\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " Nov 03 22:52:13.437587 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437526 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-data\") pod \"d8e787c8-9530-4516-ac62-2de3e352fdab\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " Nov 03 22:52:13.437879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437602 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-envoy\") pod \"d8e787c8-9530-4516-ac62-2de3e352fdab\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " Nov 03 22:52:13.437879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437631 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-certs\") pod \"d8e787c8-9530-4516-ac62-2de3e352fdab\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " Nov 03 22:52:13.437879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437652 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-socket" pod "d8e787c8-9530-4516-ac62-2de3e352fdab" (UID: "d8e787c8-9530-4516-ac62-2de3e352fdab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:13.437879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437751 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-token\" (UniqueName: \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-token\") pod \"d8e787c8-9530-4516-ac62-2de3e352fdab\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " Nov 03 22:52:13.437879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437820 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-credential-socket\") pod \"d8e787c8-9530-4516-ac62-2de3e352fdab\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " Nov 03 22:52:13.437879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437860 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-data" pod "d8e787c8-9530-4516-ac62-2de3e352fdab" (UID: "d8e787c8-9530-4516-ac62-2de3e352fdab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:13.438195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437878 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-certs" pod "d8e787c8-9530-4516-ac62-2de3e352fdab" (UID: "d8e787c8-9530-4516-ac62-2de3e352fdab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:13.438195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.437943 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e787c8-9530-4516-ac62-2de3e352fdab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e787c8-9530-4516-ac62-2de3e352fdab-istiod-ca-cert\") pod \"d8e787c8-9530-4516-ac62-2de3e352fdab\" (UID: \"d8e787c8-9530-4516-ac62-2de3e352fdab\") " Nov 03 22:52:13.438195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.438148 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-credential-socket" pod "d8e787c8-9530-4516-ac62-2de3e352fdab" (UID: "d8e787c8-9530-4516-ac62-2de3e352fdab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:13.438345 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.438271 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e787c8-9530-4516-ac62-2de3e352fdab-istiod-ca-cert" pod "d8e787c8-9530-4516-ac62-2de3e352fdab" (UID: "d8e787c8-9530-4516-ac62-2de3e352fdab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:13.438590 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.438571 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:13.438641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.438599 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e787c8-9530-4516-ac62-2de3e352fdab-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:13.438641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.438614 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:13.438641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.438629 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:13.438768 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.438645 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:13.439837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.439816 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-envoy" pod "d8e787c8-9530-4516-ac62-2de3e352fdab" (UID: "d8e787c8-9530-4516-ac62-2de3e352fdab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:13.439929 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.439858 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-podinfo" pod "d8e787c8-9530-4516-ac62-2de3e352fdab" (UID: "d8e787c8-9530-4516-ac62-2de3e352fdab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:13.440111 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.440089 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-token" pod "d8e787c8-9530-4516-ac62-2de3e352fdab" (UID: "d8e787c8-9530-4516-ac62-2de3e352fdab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:13.440223 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.440113 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-kube-api-access-bjx79" pod "d8e787c8-9530-4516-ac62-2de3e352fdab" (UID: "d8e787c8-9530-4516-ac62-2de3e352fdab"). InnerVolumeSpecName "kube-api-access-bjx79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:13.539955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.539864 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:13.539955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.539901 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjx79\" (UniqueName: \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-kube-api-access-bjx79\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:13.539955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.539912 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:13.539955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.539922 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e787c8-9530-4516-ac62-2de3e352fdab-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:13.596242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.596201 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph"] Nov 03 22:52:13.612942 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:13.611820 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26"] Nov 03 22:52:14.011244 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.011207 6234 generic.go:356] "Generic (PLEG): container finished" podID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerID="d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c" exitCode=0 Nov 03 22:52:14.011666 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.011261 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" Nov 03 22:52:14.011666 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.011290 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" event={"ID":"d8e787c8-9530-4516-ac62-2de3e352fdab","Type":"ContainerDied","Data":"d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c"} Nov 03 22:52:14.011666 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.011348 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn" event={"ID":"d8e787c8-9530-4516-ac62-2de3e352fdab","Type":"ContainerDied","Data":"2e7e3d99d32481eafaff95b368390ab93ca310b40e3ae8bad1f55dacd5d12bbe"} Nov 03 22:52:14.011666 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.011374 6234 scope.go:122] "RemoveContainer" containerID="d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c" Nov 03 22:52:14.021427 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.021405 6234 scope.go:122] "RemoveContainer" containerID="186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07" Nov 03 22:52:14.030168 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.030121 6234 scope.go:122] "RemoveContainer" containerID="d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c" Nov 03 22:52:14.030420 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:14.030398 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c\": container with ID starting with d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c not found: ID does not exist" containerID="d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c" Nov 03 22:52:14.030469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.030430 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c"} err="failed to get container status \"d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c\": rpc error: code = NotFound desc = could not find container \"d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c\": container with ID starting with d3afff658d4888bb4b3ef97725b33f772366119e9dccb6aff82217d142acff3c not found: ID does not exist" Nov 03 22:52:14.030469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.030448 6234 scope.go:122] "RemoveContainer" containerID="186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07" Nov 03 22:52:14.030767 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:14.030748 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07\": container with ID starting with 186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07 not found: ID does not exist" containerID="186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07" Nov 03 22:52:14.030809 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.030775 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07"} err="failed to get container status \"186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07\": rpc error: code = NotFound desc = could not find container \"186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07\": container with ID starting with 186c5e65ee0869ee6f2d392ea221d9575039e7342e3bcf3118dc9d985812fa07 not found: ID does not exist" Nov 03 22:52:14.164960 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.164922 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn"] Nov 03 22:52:14.171346 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.171318 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-472b46-openshift-default-5696968f69-wkvhn"] Nov 03 22:52:14.704607 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.704577 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" path="/var/lib/kubelet/pods/57101afc-e8db-439a-b8fe-ba005d025ea3/volumes" Nov 03 22:52:14.705023 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:14.705010 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" path="/var/lib/kubelet/pods/d8e787c8-9530-4516-ac62-2de3e352fdab/volumes" Nov 03 22:52:25.471837 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:25.471798 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd"] Nov 03 22:52:25.518409 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:25.518375 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm"] Nov 03 22:52:27.723487 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:27.723458 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:52:28.619623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.619580 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" containerID="cri-o://27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce" gracePeriod=45 Nov 03 22:52:28.632220 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.632178 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" containerID="cri-o://a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f" gracePeriod=45 Nov 03 22:52:28.892879 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.892854 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:52:28.912477 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.912454 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:52:28.922348 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922318 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-dshm\") pod \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " Nov 03 22:52:28.922466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922404 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-dshm\") pod \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " Nov 03 22:52:28.922466 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922459 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-model-cache\") pod \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " Nov 03 22:52:28.922569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922488 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tmp-dir\") pod \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " Nov 03 22:52:28.922569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922543 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb140cc5-3a6c-4b91-aadc-d803c46a9374-kube-api-access-xz9tm\" (UniqueName: \"kubernetes.io/projected/bb140cc5-3a6c-4b91-aadc-d803c46a9374-kube-api-access-xz9tm\") pod \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " Nov 03 22:52:28.922684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922622 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-home\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-home\") pod \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " Nov 03 22:52:28.922743 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922688 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-home\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-home\") pod \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " Nov 03 22:52:28.922799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922715 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-model-cache" pod "bb140cc5-3a6c-4b91-aadc-d803c46a9374" (UID: "bb140cc5-3a6c-4b91-aadc-d803c46a9374"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:28.922799 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922728 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tls-certs\") pod \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " Nov 03 22:52:28.922906 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922801 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tmp-dir" pod "ec6adb50-5a16-44e8-9f09-a45b32cafd1f" (UID: "ec6adb50-5a16-44e8-9f09-a45b32cafd1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:28.922906 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922810 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-model-cache\") pod \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " Nov 03 22:52:28.923018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922887 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-home" pod "ec6adb50-5a16-44e8-9f09-a45b32cafd1f" (UID: "ec6adb50-5a16-44e8-9f09-a45b32cafd1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:28.923018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922922 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tmp-dir\") pod \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\" (UID: \"bb140cc5-3a6c-4b91-aadc-d803c46a9374\") " Nov 03 22:52:28.923018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922968 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tls-certs\") pod \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " Nov 03 22:52:28.923018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922978 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-model-cache" pod "ec6adb50-5a16-44e8-9f09-a45b32cafd1f" (UID: "ec6adb50-5a16-44e8-9f09-a45b32cafd1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:28.923018 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.922977 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-home" pod "bb140cc5-3a6c-4b91-aadc-d803c46a9374" (UID: "bb140cc5-3a6c-4b91-aadc-d803c46a9374"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:28.923264 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.923018 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-kube-api-access-lswkh\" (UniqueName: \"kubernetes.io/projected/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-kube-api-access-lswkh\") pod \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\" (UID: \"ec6adb50-5a16-44e8-9f09-a45b32cafd1f\") " Nov 03 22:52:28.923264 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.923138 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tmp-dir" pod "bb140cc5-3a6c-4b91-aadc-d803c46a9374" (UID: "bb140cc5-3a6c-4b91-aadc-d803c46a9374"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:28.923564 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.923546 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:28.923656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.923567 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:28.923656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.923582 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:28.923656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.923596 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:28.923656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.923609 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:28.923656 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.923623 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:28.925406 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.925383 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-dshm" pod "ec6adb50-5a16-44e8-9f09-a45b32cafd1f" (UID: "ec6adb50-5a16-44e8-9f09-a45b32cafd1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:28.925547 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.925526 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tls-certs" pod "bb140cc5-3a6c-4b91-aadc-d803c46a9374" (UID: "bb140cc5-3a6c-4b91-aadc-d803c46a9374"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:28.925623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.925544 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb140cc5-3a6c-4b91-aadc-d803c46a9374-kube-api-access-xz9tm" pod "bb140cc5-3a6c-4b91-aadc-d803c46a9374" (UID: "bb140cc5-3a6c-4b91-aadc-d803c46a9374"). InnerVolumeSpecName "kube-api-access-xz9tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:28.925623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.925606 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-kube-api-access-lswkh" pod "ec6adb50-5a16-44e8-9f09-a45b32cafd1f" (UID: "ec6adb50-5a16-44e8-9f09-a45b32cafd1f"). InnerVolumeSpecName "kube-api-access-lswkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:28.925736 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.925695 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-dshm" pod "bb140cc5-3a6c-4b91-aadc-d803c46a9374" (UID: "bb140cc5-3a6c-4b91-aadc-d803c46a9374"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:28.926783 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:28.926759 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tls-certs" pod "ec6adb50-5a16-44e8-9f09-a45b32cafd1f" (UID: "ec6adb50-5a16-44e8-9f09-a45b32cafd1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:29.001440 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:29.001399 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:52:29.025182 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.025147 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb140cc5-3a6c-4b91-aadc-d803c46a9374-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.025182 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.025184 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.025369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.025202 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz9tm\" (UniqueName: \"kubernetes.io/projected/bb140cc5-3a6c-4b91-aadc-d803c46a9374-kube-api-access-xz9tm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.025369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.025219 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb140cc5-3a6c-4b91-aadc-d803c46a9374-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.025369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.025241 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.025369 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.025256 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lswkh\" (UniqueName: \"kubernetes.io/projected/ec6adb50-5a16-44e8-9f09-a45b32cafd1f-kube-api-access-lswkh\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.118727 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.118688 6234 generic.go:356] "Generic (PLEG): container finished" podID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerID="a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f" exitCode=0 Nov 03 22:52:29.118906 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.118865 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" event={"ID":"ec6adb50-5a16-44e8-9f09-a45b32cafd1f","Type":"ContainerDied","Data":"a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f"} Nov 03 22:52:29.118906 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.118894 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" event={"ID":"ec6adb50-5a16-44e8-9f09-a45b32cafd1f","Type":"ContainerDied","Data":"4a6e52e5561cfb2bd3b5d7453cacca5d6f24edb268b2689e5c7e3730eb9fa4d4"} Nov 03 22:52:29.119025 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.118909 6234 scope.go:122] "RemoveContainer" containerID="a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f" Nov 03 22:52:29.119025 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.119022 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" Nov 03 22:52:29.127914 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.127878 6234 generic.go:356] "Generic (PLEG): container finished" podID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerID="27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce" exitCode=0 Nov 03 22:52:29.128067 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.127944 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" event={"ID":"bb140cc5-3a6c-4b91-aadc-d803c46a9374","Type":"ContainerDied","Data":"27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce"} Nov 03 22:52:29.128067 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.127982 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" event={"ID":"bb140cc5-3a6c-4b91-aadc-d803c46a9374","Type":"ContainerDied","Data":"93f91026c766e8def891b99d6ea61c1608611e3d933f869c44b1cc45a69eb9e3"} Nov 03 22:52:29.128200 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.128085 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph" Nov 03 22:52:29.139924 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.139896 6234 scope.go:122] "RemoveContainer" containerID="a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f" Nov 03 22:52:29.140403 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:29.140352 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f\": container with ID starting with a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f not found: ID does not exist" containerID="a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f" Nov 03 22:52:29.140535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.140400 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f"} err="failed to get container status \"a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f\": rpc error: code = NotFound desc = could not find container \"a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f\": container with ID starting with a98aa497cef9187e25f30950eaaca7156309325fb5aba7bc29704763e3184c7f not found: ID does not exist" Nov 03 22:52:29.140535 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.140425 6234 scope.go:122] "RemoveContainer" containerID="27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce" Nov 03 22:52:29.154677 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.154651 6234 scope.go:122] "RemoveContainer" containerID="27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce" Nov 03 22:52:29.155021 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:29.154995 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce\": container with ID starting with 27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce not found: ID does not exist" containerID="27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce" Nov 03 22:52:29.155112 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.155034 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce"} err="failed to get container status \"27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce\": rpc error: code = NotFound desc = could not find container \"27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce\": container with ID starting with 27c9068458ff0ae5aec929107c98cb7ff82a034cc294f37ad5840480604873ce not found: ID does not exist" Nov 03 22:52:29.291199 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.288296 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26"] Nov 03 22:52:29.296873 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.296789 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26"] Nov 03 22:52:29.400086 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.400004 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph"] Nov 03 22:52:29.404368 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.404341 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/premium-w4-f7e6db-kserve-76c4ff9964-hc2ph"] Nov 03 22:52:29.889430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:29.889386 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/simulator-w4-f7e6db-kserve-6587df6689-s8j26" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:52:30.704259 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:30.704216 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" path="/var/lib/kubelet/pods/bb140cc5-3a6c-4b91-aadc-d803c46a9374/volumes" Nov 03 22:52:30.704830 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:30.704685 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" path="/var/lib/kubelet/pods/ec6adb50-5a16-44e8-9f09-a45b32cafd1f/volumes" Nov 03 22:52:32.516783 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:32.516744 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks"] Nov 03 22:52:37.517191 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:37.517144 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" podUID="735f20bd-8a1e-4c65-ac98-6bf603a6ecfa" containerName="payload-processing" containerID="cri-o://38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290" gracePeriod=25 Nov 03 22:52:37.794340 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:37.794313 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:52:37.943766 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:37.943736 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-plugins-config-volume\") pod \"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa\" (UID: \"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa\") " Nov 03 22:52:37.943932 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:37.943854 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-kube-api-access-xst5w\" (UniqueName: \"kubernetes.io/projected/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-kube-api-access-xst5w\") pod \"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa\" (UID: \"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa\") " Nov 03 22:52:37.944219 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:37.944191 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-plugins-config-volume" pod "735f20bd-8a1e-4c65-ac98-6bf603a6ecfa" (UID: "735f20bd-8a1e-4c65-ac98-6bf603a6ecfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:37.946024 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:37.946000 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-kube-api-access-xst5w" pod "735f20bd-8a1e-4c65-ac98-6bf603a6ecfa" (UID: "735f20bd-8a1e-4c65-ac98-6bf603a6ecfa"). InnerVolumeSpecName "kube-api-access-xst5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:38.045215 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.045140 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xst5w\" (UniqueName: \"kubernetes.io/projected/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-kube-api-access-xst5w\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:38.045215 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.045169 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:38.187235 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.187198 6234 generic.go:356] "Generic (PLEG): container finished" podID="735f20bd-8a1e-4c65-ac98-6bf603a6ecfa" containerID="38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290" exitCode=0 Nov 03 22:52:38.187419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.187244 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" Nov 03 22:52:38.187419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.187280 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" event={"ID":"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa","Type":"ContainerDied","Data":"38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290"} Nov 03 22:52:38.187419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.187336 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks" event={"ID":"735f20bd-8a1e-4c65-ac98-6bf603a6ecfa","Type":"ContainerDied","Data":"3dc697ee4e4ebcc2dd0e8e7f24efcb8b81c0fecd42685d36527075bfcbe4228c"} Nov 03 22:52:38.187419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.187359 6234 scope.go:122] "RemoveContainer" containerID="38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290" Nov 03 22:52:38.197303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.197283 6234 scope.go:122] "RemoveContainer" containerID="38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290" Nov 03 22:52:38.197575 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:38.197554 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290\": container with ID starting with 38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290 not found: ID does not exist" containerID="38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290" Nov 03 22:52:38.197646 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.197583 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290"} err="failed to get container status \"38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290\": rpc error: code = NotFound desc = could not find container \"38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290\": container with ID starting with 38fdcc43f11b8ae1abe371f84098d389c87479989c096520c24526b77c7f1290 not found: ID does not exist" Nov 03 22:52:38.340315 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.340230 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks"] Nov 03 22:52:38.386851 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.386814 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-906673-59486f8964-dbgks"] Nov 03 22:52:38.704847 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:38.704815 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735f20bd-8a1e-4c65-ac98-6bf603a6ecfa" path="/var/lib/kubelet/pods/735f20bd-8a1e-4c65-ac98-6bf603a6ecfa/volumes" Nov 03 22:52:39.723066 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:39.723032 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865"] Nov 03 22:52:39.723530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:39.723255 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" podUID="baf62f19-94ba-4650-80a5-f19383286180" containerName="maas-api" containerID="cri-o://9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463" gracePeriod=30 Nov 03 22:52:39.999641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:39.999611 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:52:40.068748 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.068716 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf62f19-94ba-4650-80a5-f19383286180-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf62f19-94ba-4650-80a5-f19383286180-openshift-service-ca\") pod \"baf62f19-94ba-4650-80a5-f19383286180\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " Nov 03 22:52:40.068929 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.068769 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf62f19-94ba-4650-80a5-f19383286180-kube-api-access-hnzr4\" (UniqueName: \"kubernetes.io/projected/baf62f19-94ba-4650-80a5-f19383286180-kube-api-access-hnzr4\") pod \"baf62f19-94ba-4650-80a5-f19383286180\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " Nov 03 22:52:40.068929 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.068828 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-metrics-certs\") pod \"baf62f19-94ba-4650-80a5-f19383286180\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " Nov 03 22:52:40.068929 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.068847 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-maas-api-tls\") pod \"baf62f19-94ba-4650-80a5-f19383286180\" (UID: \"baf62f19-94ba-4650-80a5-f19383286180\") " Nov 03 22:52:40.069120 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.069093 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf62f19-94ba-4650-80a5-f19383286180-openshift-service-ca" pod "baf62f19-94ba-4650-80a5-f19383286180" (UID: "baf62f19-94ba-4650-80a5-f19383286180"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:40.069275 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.069250 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf62f19-94ba-4650-80a5-f19383286180-openshift-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.070994 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.070969 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-metrics-certs" pod "baf62f19-94ba-4650-80a5-f19383286180" (UID: "baf62f19-94ba-4650-80a5-f19383286180"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:40.071105 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.071003 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-maas-api-tls" pod "baf62f19-94ba-4650-80a5-f19383286180" (UID: "baf62f19-94ba-4650-80a5-f19383286180"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:40.071105 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.071085 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf62f19-94ba-4650-80a5-f19383286180-kube-api-access-hnzr4" pod "baf62f19-94ba-4650-80a5-f19383286180" (UID: "baf62f19-94ba-4650-80a5-f19383286180"). InnerVolumeSpecName "kube-api-access-hnzr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:40.170581 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.170460 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-metrics-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.170581 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.170524 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf62f19-94ba-4650-80a5-f19383286180-maas-api-tls\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.170581 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.170542 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnzr4\" (UniqueName: \"kubernetes.io/projected/baf62f19-94ba-4650-80a5-f19383286180-kube-api-access-hnzr4\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.200013 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.199982 6234 generic.go:356] "Generic (PLEG): container finished" podID="baf62f19-94ba-4650-80a5-f19383286180" containerID="9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463" exitCode=0 Nov 03 22:52:40.200185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.200029 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" Nov 03 22:52:40.200185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.200068 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" event={"ID":"baf62f19-94ba-4650-80a5-f19383286180","Type":"ContainerDied","Data":"9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463"} Nov 03 22:52:40.200185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.200127 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865" event={"ID":"baf62f19-94ba-4650-80a5-f19383286180","Type":"ContainerDied","Data":"fc4f5e5ccddb91d297d19a17ccd44c791e3b418468a64d29cb55a04f42c36854"} Nov 03 22:52:40.200185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.200159 6234 scope.go:122] "RemoveContainer" containerID="9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463" Nov 03 22:52:40.210082 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.210064 6234 scope.go:122] "RemoveContainer" containerID="9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463" Nov 03 22:52:40.210358 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:40.210303 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463\": container with ID starting with 9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463 not found: ID does not exist" containerID="9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463" Nov 03 22:52:40.210419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.210367 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463"} err="failed to get container status \"9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463\": rpc error: code = NotFound desc = could not find container \"9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463\": container with ID starting with 9beef34ba765016988b9ce13aec8ef83870b06af986b19962f2fa31ce5688463 not found: ID does not exist" Nov 03 22:52:40.352966 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.352929 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865"] Nov 03 22:52:40.356393 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.356368 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41bb2b-c75d6c7-mj865"] Nov 03 22:52:40.492706 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.492664 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" containerID="cri-o://00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91" gracePeriod=45 Nov 03 22:52:40.538447 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.538396 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" containerID="cri-o://f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b" gracePeriod=45 Nov 03 22:52:40.719349 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.719314 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf62f19-94ba-4650-80a5-f19383286180" path="/var/lib/kubelet/pods/baf62f19-94ba-4650-80a5-f19383286180/volumes" Nov 03 22:52:40.843429 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.843400 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:52:40.874926 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.874903 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:52:40.979652 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.979621 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-model-cache\") pod \"8dbbd364-d950-4467-85d3-fa4d0740adb8\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " Nov 03 22:52:40.979838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.979708 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-tmp-dir\") pod \"31b4fadd-1467-4cab-991b-08a56498eeae\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " Nov 03 22:52:40.979838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.979761 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dbbd364-d950-4467-85d3-fa4d0740adb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbbd364-d950-4467-85d3-fa4d0740adb8-tls-certs\") pod \"8dbbd364-d950-4467-85d3-fa4d0740adb8\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " Nov 03 22:52:40.979838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.979795 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-tmp-dir\") pod \"8dbbd364-d950-4467-85d3-fa4d0740adb8\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " Nov 03 22:52:40.979838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.979833 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-home\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-home\") pod \"31b4fadd-1467-4cab-991b-08a56498eeae\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " Nov 03 22:52:40.980052 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.979934 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-model-cache" pod "8dbbd364-d950-4467-85d3-fa4d0740adb8" (UID: "8dbbd364-d950-4467-85d3-fa4d0740adb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.980052 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.979971 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-tmp-dir" pod "31b4fadd-1467-4cab-991b-08a56498eeae" (UID: "31b4fadd-1467-4cab-991b-08a56498eeae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.980052 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.979988 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b4fadd-1467-4cab-991b-08a56498eeae-kube-api-access-8jxtm\" (UniqueName: \"kubernetes.io/projected/31b4fadd-1467-4cab-991b-08a56498eeae-kube-api-access-8jxtm\") pod \"31b4fadd-1467-4cab-991b-08a56498eeae\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " Nov 03 22:52:40.980208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980074 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-tmp-dir" pod "8dbbd364-d950-4467-85d3-fa4d0740adb8" (UID: "8dbbd364-d950-4467-85d3-fa4d0740adb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.980208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980086 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-dshm\") pod \"8dbbd364-d950-4467-85d3-fa4d0740adb8\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " Nov 03 22:52:40.980208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980078 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-home" pod "31b4fadd-1467-4cab-991b-08a56498eeae" (UID: "31b4fadd-1467-4cab-991b-08a56498eeae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.980208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980118 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31b4fadd-1467-4cab-991b-08a56498eeae-tls-certs\" (UniqueName: \"kubernetes.io/secret/31b4fadd-1467-4cab-991b-08a56498eeae-tls-certs\") pod \"31b4fadd-1467-4cab-991b-08a56498eeae\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " Nov 03 22:52:40.980208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980152 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-home\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-home\") pod \"8dbbd364-d950-4467-85d3-fa4d0740adb8\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " Nov 03 22:52:40.980208 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980197 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-dshm\") pod \"31b4fadd-1467-4cab-991b-08a56498eeae\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " Nov 03 22:52:40.980488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980242 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-model-cache\") pod \"31b4fadd-1467-4cab-991b-08a56498eeae\" (UID: \"31b4fadd-1467-4cab-991b-08a56498eeae\") " Nov 03 22:52:40.980488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980275 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbbd364-d950-4467-85d3-fa4d0740adb8-kube-api-access-s4fnv\" (UniqueName: \"kubernetes.io/projected/8dbbd364-d950-4467-85d3-fa4d0740adb8-kube-api-access-s4fnv\") pod \"8dbbd364-d950-4467-85d3-fa4d0740adb8\" (UID: \"8dbbd364-d950-4467-85d3-fa4d0740adb8\") " Nov 03 22:52:40.980488 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980422 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-home" pod "8dbbd364-d950-4467-85d3-fa4d0740adb8" (UID: "8dbbd364-d950-4467-85d3-fa4d0740adb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.980832 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980809 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.980978 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980836 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.980978 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980848 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.980978 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980862 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.980978 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.980874 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:40.981421 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.981397 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-model-cache" pod "31b4fadd-1467-4cab-991b-08a56498eeae" (UID: "31b4fadd-1467-4cab-991b-08a56498eeae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.984565 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.984542 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b4fadd-1467-4cab-991b-08a56498eeae-kube-api-access-8jxtm" pod "31b4fadd-1467-4cab-991b-08a56498eeae" (UID: "31b4fadd-1467-4cab-991b-08a56498eeae"). InnerVolumeSpecName "kube-api-access-8jxtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:40.984696 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.984564 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbbd364-d950-4467-85d3-fa4d0740adb8-kube-api-access-s4fnv" pod "8dbbd364-d950-4467-85d3-fa4d0740adb8" (UID: "8dbbd364-d950-4467-85d3-fa4d0740adb8"). InnerVolumeSpecName "kube-api-access-s4fnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:40.984696 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.984575 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31b4fadd-1467-4cab-991b-08a56498eeae-tls-certs" pod "31b4fadd-1467-4cab-991b-08a56498eeae" (UID: "31b4fadd-1467-4cab-991b-08a56498eeae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:40.984696 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.984685 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dbbd364-d950-4467-85d3-fa4d0740adb8-tls-certs" pod "8dbbd364-d950-4467-85d3-fa4d0740adb8" (UID: "8dbbd364-d950-4467-85d3-fa4d0740adb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:40.984805 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.984727 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-dshm" pod "8dbbd364-d950-4467-85d3-fa4d0740adb8" (UID: "8dbbd364-d950-4467-85d3-fa4d0740adb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:40.984805 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:40.984740 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-dshm" pod "31b4fadd-1467-4cab-991b-08a56498eeae" (UID: "31b4fadd-1467-4cab-991b-08a56498eeae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:41.082290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.082201 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.082290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.082237 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b4fadd-1467-4cab-991b-08a56498eeae-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.082290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.082249 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4fnv\" (UniqueName: \"kubernetes.io/projected/8dbbd364-d950-4467-85d3-fa4d0740adb8-kube-api-access-s4fnv\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.082290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.082259 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbbd364-d950-4467-85d3-fa4d0740adb8-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.082290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.082267 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jxtm\" (UniqueName: \"kubernetes.io/projected/31b4fadd-1467-4cab-991b-08a56498eeae-kube-api-access-8jxtm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.082290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.082277 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dbbd364-d950-4467-85d3-fa4d0740adb8-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.082290 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.082290 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b4fadd-1467-4cab-991b-08a56498eeae-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:41.207861 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.207823 6234 generic.go:356] "Generic (PLEG): container finished" podID="31b4fadd-1467-4cab-991b-08a56498eeae" containerID="00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91" exitCode=0 Nov 03 22:52:41.208059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.207870 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" Nov 03 22:52:41.208059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.207906 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" event={"ID":"31b4fadd-1467-4cab-991b-08a56498eeae","Type":"ContainerDied","Data":"00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91"} Nov 03 22:52:41.208059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.207954 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" event={"ID":"31b4fadd-1467-4cab-991b-08a56498eeae","Type":"ContainerDied","Data":"4d7e2abde2c2b9223958af157915b727f3087ca81e9e3a406f260be3328e342b"} Nov 03 22:52:41.208059 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.207977 6234 scope.go:122] "RemoveContainer" containerID="00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91" Nov 03 22:52:41.209484 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.209459 6234 generic.go:356] "Generic (PLEG): container finished" podID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerID="f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b" exitCode=0 Nov 03 22:52:41.209620 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.209530 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" Nov 03 22:52:41.209620 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.209526 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" event={"ID":"8dbbd364-d950-4467-85d3-fa4d0740adb8","Type":"ContainerDied","Data":"f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b"} Nov 03 22:52:41.209693 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.209634 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm" event={"ID":"8dbbd364-d950-4467-85d3-fa4d0740adb8","Type":"ContainerDied","Data":"8830c4bbf8fd9b26ffaa4a353e0cad5d90a296bac2fd218c04d221c15f9d2142"} Nov 03 22:52:41.217251 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.217229 6234 scope.go:122] "RemoveContainer" containerID="00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91" Nov 03 22:52:41.217585 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:41.217564 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91\": container with ID starting with 00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91 not found: ID does not exist" containerID="00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91" Nov 03 22:52:41.217684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.217595 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91"} err="failed to get container status \"00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91\": rpc error: code = NotFound desc = could not find container \"00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91\": container with ID starting with 00c70c38fb857807aea86c09866191027cc8350c049ad27962035b181a063c91 not found: ID does not exist" Nov 03 22:52:41.217684 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.217618 6234 scope.go:122] "RemoveContainer" containerID="f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b" Nov 03 22:52:41.225548 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.225531 6234 scope.go:122] "RemoveContainer" containerID="f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b" Nov 03 22:52:41.225790 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:41.225774 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b\": container with ID starting with f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b not found: ID does not exist" containerID="f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b" Nov 03 22:52:41.225843 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.225796 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b"} err="failed to get container status \"f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b\": rpc error: code = NotFound desc = could not find container \"f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b\": container with ID starting with f62e661a50d43845a2600e625eb13cb7af990bc01e1c8f6fbd3fba23b538792b not found: ID does not exist" Nov 03 22:52:41.360451 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.360379 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm"] Nov 03 22:52:41.364271 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.364242 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-w2-906673-kserve-66c87876-kkmzm"] Nov 03 22:52:41.465404 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.465372 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd"] Nov 03 22:52:41.469426 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.469403 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd"] Nov 03 22:52:41.817978 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:41.817928 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-906673/distinct-2-w2-906673-kserve-85665659c5-6gdvd" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 22:52:42.703609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:42.703577 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" path="/var/lib/kubelet/pods/31b4fadd-1467-4cab-991b-08a56498eeae/volumes" Nov 03 22:52:42.704035 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:42.704021 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" path="/var/lib/kubelet/pods/8dbbd364-d950-4467-85d3-fa4d0740adb8/volumes" Nov 03 22:52:48.965319 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:48.965286 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8"] Nov 03 22:52:48.965731 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:48.965535 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerName="istio-proxy" containerID="cri-o://157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b" gracePeriod=30 Nov 03 22:52:54.252562 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.246517 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:52:54.287195 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.287162 6234 generic.go:356] "Generic (PLEG): container finished" podID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerID="157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b" exitCode=0 Nov 03 22:52:54.287348 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.287249 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" Nov 03 22:52:54.287348 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.287256 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" event={"ID":"a7a2bf96-b555-484f-b589-f3e68ab2878d","Type":"ContainerDied","Data":"157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b"} Nov 03 22:52:54.287425 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.287357 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8" event={"ID":"a7a2bf96-b555-484f-b589-f3e68ab2878d","Type":"ContainerDied","Data":"9c0615b2090f47997376344aedb79a335ce584c708fdc59fea3c3431b55f5ea7"} Nov 03 22:52:54.287425 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.287392 6234 scope.go:122] "RemoveContainer" containerID="157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b" Nov 03 22:52:54.298110 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.298092 6234 scope.go:122] "RemoveContainer" containerID="157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b" Nov 03 22:52:54.298367 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:54.298345 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b\": container with ID starting with 157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b not found: ID does not exist" containerID="157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b" Nov 03 22:52:54.298451 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.298379 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b"} err="failed to get container status \"157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b\": rpc error: code = NotFound desc = could not find container \"157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b\": container with ID starting with 157e33a050e0d65125738aa9447eb237c599a59f7188288d9b9abc87ff07af2b not found: ID does not exist" Nov 03 22:52:54.328887 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.328857 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-credential-socket\") pod \"a7a2bf96-b555-484f-b589-f3e68ab2878d\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " Nov 03 22:52:54.329026 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.328903 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-envoy\") pod \"a7a2bf96-b555-484f-b589-f3e68ab2878d\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " Nov 03 22:52:54.329026 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.328954 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-kube-api-access-4p2vv\" (UniqueName: \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-kube-api-access-4p2vv\") pod \"a7a2bf96-b555-484f-b589-f3e68ab2878d\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " Nov 03 22:52:54.329026 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.328991 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-certs\") pod \"a7a2bf96-b555-484f-b589-f3e68ab2878d\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " Nov 03 22:52:54.329185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329034 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-token\" (UniqueName: \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-token\") pod \"a7a2bf96-b555-484f-b589-f3e68ab2878d\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " Nov 03 22:52:54.329185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329069 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-socket\") pod \"a7a2bf96-b555-484f-b589-f3e68ab2878d\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " Nov 03 22:52:54.329185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329126 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-podinfo\") pod \"a7a2bf96-b555-484f-b589-f3e68ab2878d\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " Nov 03 22:52:54.329185 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329160 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a2bf96-b555-484f-b589-f3e68ab2878d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2bf96-b555-484f-b589-f3e68ab2878d-istiod-ca-cert\") pod \"a7a2bf96-b555-484f-b589-f3e68ab2878d\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " Nov 03 22:52:54.329387 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329176 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-credential-socket" pod "a7a2bf96-b555-484f-b589-f3e68ab2878d" (UID: "a7a2bf96-b555-484f-b589-f3e68ab2878d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:54.329387 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329236 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-data\") pod \"a7a2bf96-b555-484f-b589-f3e68ab2878d\" (UID: \"a7a2bf96-b555-484f-b589-f3e68ab2878d\") " Nov 03 22:52:54.329485 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329376 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-socket" pod "a7a2bf96-b555-484f-b589-f3e68ab2878d" (UID: "a7a2bf96-b555-484f-b589-f3e68ab2878d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:54.329955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329704 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-certs" pod "a7a2bf96-b555-484f-b589-f3e68ab2878d" (UID: "a7a2bf96-b555-484f-b589-f3e68ab2878d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:54.329955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329733 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-data" pod "a7a2bf96-b555-484f-b589-f3e68ab2878d" (UID: "a7a2bf96-b555-484f-b589-f3e68ab2878d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:54.329955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329716 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.329955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329807 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.329955 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.329918 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a2bf96-b555-484f-b589-f3e68ab2878d-istiod-ca-cert" pod "a7a2bf96-b555-484f-b589-f3e68ab2878d" (UID: "a7a2bf96-b555-484f-b589-f3e68ab2878d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:54.331429 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.331403 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-envoy" pod "a7a2bf96-b555-484f-b589-f3e68ab2878d" (UID: "a7a2bf96-b555-484f-b589-f3e68ab2878d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:54.331809 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.331782 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-kube-api-access-4p2vv" pod "a7a2bf96-b555-484f-b589-f3e68ab2878d" (UID: "a7a2bf96-b555-484f-b589-f3e68ab2878d"). InnerVolumeSpecName "kube-api-access-4p2vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:54.331941 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.331903 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-podinfo" pod "a7a2bf96-b555-484f-b589-f3e68ab2878d" (UID: "a7a2bf96-b555-484f-b589-f3e68ab2878d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:54.332854 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.332830 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-token" pod "a7a2bf96-b555-484f-b589-f3e68ab2878d" (UID: "a7a2bf96-b555-484f-b589-f3e68ab2878d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:54.431433 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.431395 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.431433 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.431436 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p2vv\" (UniqueName: \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-kube-api-access-4p2vv\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.431803 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.431454 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.431803 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.431470 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.431803 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.431484 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.431803 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.431513 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2bf96-b555-484f-b589-f3e68ab2878d-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.431803 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.431531 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a2bf96-b555-484f-b589-f3e68ab2878d-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:54.708083 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.708049 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8"] Nov 03 22:52:54.715507 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:54.715463 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-rrsc8"] Nov 03 22:52:56.705759 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:52:56.705717 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" path="/var/lib/kubelet/pods/a7a2bf96-b555-484f-b589-f3e68ab2878d/volumes" Nov 03 22:52:58.995435 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:52:58.995397 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:53:02.591718 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.591685 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl"] Nov 03 22:53:02.592206 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.591876 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" podUID="2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" containerName="discovery" containerID="cri-o://7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe" gracePeriod=30 Nov 03 22:53:02.868792 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.868766 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:53:02.943928 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.943895 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-local-certs\") pod \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " Nov 03 22:53:02.944116 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.944026 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-cacerts\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-cacerts\") pod \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " Nov 03 22:53:02.944116 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.944081 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-ca-configmap\") pod \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " Nov 03 22:53:02.944116 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.944103 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-dns-cert\") pod \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " Nov 03 22:53:02.944312 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.944132 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-kube-api-access-6fm9s\" (UniqueName: \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-kube-api-access-6fm9s\") pod \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " Nov 03 22:53:02.944312 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.944185 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-kubeconfig\") pod \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " Nov 03 22:53:02.944312 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.944221 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-token\" (UniqueName: \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-token\") pod \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\" (UID: \"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f\") " Nov 03 22:53:02.945123 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.945070 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-ca-configmap" pod "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" (UID: "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:02.946951 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.946924 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-local-certs" pod "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" (UID: "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.947225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.947154 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-cacerts" pod "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" (UID: "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:02.947225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.947190 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-token" pod "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" (UID: "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.947386 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.947364 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-dns-cert" pod "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" (UID: "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:02.947452 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.947365 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-kube-api-access-6fm9s" pod "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" (UID: "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f"). InnerVolumeSpecName "kube-api-access-6fm9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.948085 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:02.948068 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-kubeconfig" pod "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" (UID: "2a4b2a00-37a7-4c3b-ae3a-a901ae71815f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:03.046437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.046398 6234 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-kubeconfig\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.046437 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.046437 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.046662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.046453 6234 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-local-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.046662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.046467 6234 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-cacerts\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.046662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.046482 6234 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-ca-configmap\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.046662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.046527 6234 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-istio-csr-dns-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.046662 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.046542 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fm9s\" (UniqueName: \"kubernetes.io/projected/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f-kube-api-access-6fm9s\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.350692 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.350646 6234 generic.go:356] "Generic (PLEG): container finished" podID="2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" containerID="7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe" exitCode=0 Nov 03 22:53:03.350892 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.350819 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" event={"ID":"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f","Type":"ContainerDied","Data":"7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe"} Nov 03 22:53:03.350956 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.350908 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" event={"ID":"2a4b2a00-37a7-4c3b-ae3a-a901ae71815f","Type":"ContainerDied","Data":"473b419ebc74dbeff5cfbed609fe8fd44c665dda5e2cf45265614ae1f32ceaed"} Nov 03 22:53:03.350956 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.350938 6234 scope.go:122] "RemoveContainer" containerID="7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe" Nov 03 22:53:03.351280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.351138 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl" Nov 03 22:53:03.362969 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.362951 6234 scope.go:122] "RemoveContainer" containerID="7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe" Nov 03 22:53:03.363293 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:53:03.363271 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe\": container with ID starting with 7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe not found: ID does not exist" containerID="7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe" Nov 03 22:53:03.363361 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.363302 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe"} err="failed to get container status \"7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe\": rpc error: code = NotFound desc = could not find container \"7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe\": container with ID starting with 7b8bc84aa048a2f7297a19bf9a57601d87e468a0ff797fa78a0bf8d6238a1bfe not found: ID does not exist" Nov 03 22:53:03.505890 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.505855 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl"] Nov 03 22:53:03.552862 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:03.552828 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fzzsl"] Nov 03 22:53:04.704048 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:04.704013 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" path="/var/lib/kubelet/pods/2a4b2a00-37a7-4c3b-ae3a-a901ae71815f/volumes" Nov 03 22:53:06.221035 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.221008 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476"] Nov 03 22:53:06.221405 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.221191 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" podUID="016058c4-df1f-4952-970d-e3f9ffe4704d" containerName="maas-api" containerID="cri-o://5438fc35e36a4feb8d3027338d8dc2b0dddd62a12365575bcf7939a6dbe637e6" gracePeriod=30 Nov 03 22:53:06.371821 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.371779 6234 generic.go:356] "Generic (PLEG): container finished" podID="016058c4-df1f-4952-970d-e3f9ffe4704d" containerID="5438fc35e36a4feb8d3027338d8dc2b0dddd62a12365575bcf7939a6dbe637e6" exitCode=0 Nov 03 22:53:06.371972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.371850 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" event={"ID":"016058c4-df1f-4952-970d-e3f9ffe4704d","Type":"ContainerDied","Data":"5438fc35e36a4feb8d3027338d8dc2b0dddd62a12365575bcf7939a6dbe637e6"} Nov 03 22:53:06.474313 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.474253 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:53:06.582326 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.582280 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-maas-api-tls\") pod \"016058c4-df1f-4952-970d-e3f9ffe4704d\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " Nov 03 22:53:06.582520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.582344 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016058c4-df1f-4952-970d-e3f9ffe4704d-kube-api-access-d6ccv\" (UniqueName: \"kubernetes.io/projected/016058c4-df1f-4952-970d-e3f9ffe4704d-kube-api-access-d6ccv\") pod \"016058c4-df1f-4952-970d-e3f9ffe4704d\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " Nov 03 22:53:06.582611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.582520 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-metrics-certs\") pod \"016058c4-df1f-4952-970d-e3f9ffe4704d\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " Nov 03 22:53:06.582611 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.582573 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016058c4-df1f-4952-970d-e3f9ffe4704d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016058c4-df1f-4952-970d-e3f9ffe4704d-openshift-service-ca\") pod \"016058c4-df1f-4952-970d-e3f9ffe4704d\" (UID: \"016058c4-df1f-4952-970d-e3f9ffe4704d\") " Nov 03 22:53:06.583248 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.583220 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016058c4-df1f-4952-970d-e3f9ffe4704d-openshift-service-ca" pod "016058c4-df1f-4952-970d-e3f9ffe4704d" (UID: "016058c4-df1f-4952-970d-e3f9ffe4704d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:06.584801 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.584756 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-maas-api-tls" pod "016058c4-df1f-4952-970d-e3f9ffe4704d" (UID: "016058c4-df1f-4952-970d-e3f9ffe4704d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:06.585088 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.585060 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016058c4-df1f-4952-970d-e3f9ffe4704d-kube-api-access-d6ccv" pod "016058c4-df1f-4952-970d-e3f9ffe4704d" (UID: "016058c4-df1f-4952-970d-e3f9ffe4704d"). InnerVolumeSpecName "kube-api-access-d6ccv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:06.585420 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.585391 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-metrics-certs" pod "016058c4-df1f-4952-970d-e3f9ffe4704d" (UID: "016058c4-df1f-4952-970d-e3f9ffe4704d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:06.683613 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.683580 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-metrics-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:06.683613 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.683606 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016058c4-df1f-4952-970d-e3f9ffe4704d-openshift-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:06.683613 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.683616 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016058c4-df1f-4952-970d-e3f9ffe4704d-maas-api-tls\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:06.683857 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:06.683624 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ccv\" (UniqueName: \"kubernetes.io/projected/016058c4-df1f-4952-970d-e3f9ffe4704d-kube-api-access-d6ccv\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:07.377419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.377385 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8"] Nov 03 22:53:07.377903 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.377605 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" podUID="7d7a8dea-f57e-40f0-8dfc-751843570fde" containerName="maas-api" containerID="cri-o://6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7" gracePeriod=30 Nov 03 22:53:07.378538 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.378490 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" event={"ID":"016058c4-df1f-4952-970d-e3f9ffe4704d","Type":"ContainerDied","Data":"5aa22b36a41144629946c497988c44a25198c706a2f32d58bc174a2fc4b0d100"} Nov 03 22:53:07.378661 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.378535 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476" Nov 03 22:53:07.378661 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.378566 6234 scope.go:122] "RemoveContainer" containerID="5438fc35e36a4feb8d3027338d8dc2b0dddd62a12365575bcf7939a6dbe637e6" Nov 03 22:53:07.591455 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.591427 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476"] Nov 03 22:53:07.595728 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.595699 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f7e6db-6fc5457cbc-n6476"] Nov 03 22:53:07.640735 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.640682 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:53:07.793698 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.793667 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7a8dea-f57e-40f0-8dfc-751843570fde-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a8dea-f57e-40f0-8dfc-751843570fde-openshift-service-ca\") pod \"7d7a8dea-f57e-40f0-8dfc-751843570fde\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " Nov 03 22:53:07.793851 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.793726 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-maas-api-tls\") pod \"7d7a8dea-f57e-40f0-8dfc-751843570fde\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " Nov 03 22:53:07.793851 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.793763 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-metrics-certs\") pod \"7d7a8dea-f57e-40f0-8dfc-751843570fde\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " Nov 03 22:53:07.793851 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.793804 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7a8dea-f57e-40f0-8dfc-751843570fde-kube-api-access-78wgf\" (UniqueName: \"kubernetes.io/projected/7d7a8dea-f57e-40f0-8dfc-751843570fde-kube-api-access-78wgf\") pod \"7d7a8dea-f57e-40f0-8dfc-751843570fde\" (UID: \"7d7a8dea-f57e-40f0-8dfc-751843570fde\") " Nov 03 22:53:07.794114 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.794084 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7a8dea-f57e-40f0-8dfc-751843570fde-openshift-service-ca" pod "7d7a8dea-f57e-40f0-8dfc-751843570fde" (UID: "7d7a8dea-f57e-40f0-8dfc-751843570fde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:07.794252 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.794235 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a8dea-f57e-40f0-8dfc-751843570fde-openshift-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:07.796088 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.796063 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-maas-api-tls" pod "7d7a8dea-f57e-40f0-8dfc-751843570fde" (UID: "7d7a8dea-f57e-40f0-8dfc-751843570fde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:07.796242 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.796219 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7a8dea-f57e-40f0-8dfc-751843570fde-kube-api-access-78wgf" pod "7d7a8dea-f57e-40f0-8dfc-751843570fde" (UID: "7d7a8dea-f57e-40f0-8dfc-751843570fde"). InnerVolumeSpecName "kube-api-access-78wgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:07.796293 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.796228 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-metrics-certs" pod "7d7a8dea-f57e-40f0-8dfc-751843570fde" (UID: "7d7a8dea-f57e-40f0-8dfc-751843570fde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:07.896157 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.896056 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-metrics-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:07.896157 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.896095 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78wgf\" (UniqueName: \"kubernetes.io/projected/7d7a8dea-f57e-40f0-8dfc-751843570fde-kube-api-access-78wgf\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:07.896157 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:07.896112 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7a8dea-f57e-40f0-8dfc-751843570fde-maas-api-tls\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:08.262335 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.262299 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp"] Nov 03 22:53:08.262572 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.262546 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" containerID="cri-o://1c89b1a59bfd32c6c7f6d3f078a8282fa675c41a24d1197104e6a842cf12ab84" gracePeriod=30 Nov 03 22:53:08.310463 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.310413 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772"] Nov 03 22:53:08.310920 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.310888 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" containerID="cri-o://c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf" gracePeriod=30 Nov 03 22:53:08.384673 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.384637 6234 generic.go:356] "Generic (PLEG): container finished" podID="7d7a8dea-f57e-40f0-8dfc-751843570fde" containerID="6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7" exitCode=0 Nov 03 22:53:08.385096 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.384684 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" event={"ID":"7d7a8dea-f57e-40f0-8dfc-751843570fde","Type":"ContainerDied","Data":"6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7"} Nov 03 22:53:08.385096 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.384693 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" Nov 03 22:53:08.385096 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.384746 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8" event={"ID":"7d7a8dea-f57e-40f0-8dfc-751843570fde","Type":"ContainerDied","Data":"1710dc6abccc4af38e6d82e9e5d7be938ac95f7e9781c1cc4532b35b9cbe8ccb"} Nov 03 22:53:08.385096 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.384776 6234 scope.go:122] "RemoveContainer" containerID="6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7" Nov 03 22:53:08.400385 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.400363 6234 scope.go:122] "RemoveContainer" containerID="6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7" Nov 03 22:53:08.400786 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:53:08.400744 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7\": container with ID starting with 6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7 not found: ID does not exist" containerID="6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7" Nov 03 22:53:08.400894 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.400796 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7"} err="failed to get container status \"6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7\": rpc error: code = NotFound desc = could not find container \"6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7\": container with ID starting with 6d761a556eb4c00543310c2017ad7ef29b58c938abff6f8d87b4379d2ad057c7 not found: ID does not exist" Nov 03 22:53:08.601044 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.600965 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8"] Nov 03 22:53:08.605905 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.605878 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-906673-66f65dcf44-j2zq8"] Nov 03 22:53:08.703375 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.703340 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016058c4-df1f-4952-970d-e3f9ffe4704d" path="/var/lib/kubelet/pods/016058c4-df1f-4952-970d-e3f9ffe4704d/volumes" Nov 03 22:53:08.703791 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:08.703777 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7a8dea-f57e-40f0-8dfc-751843570fde" path="/var/lib/kubelet/pods/7d7a8dea-f57e-40f0-8dfc-751843570fde/volumes" Nov 03 22:53:09.411114 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:09.411072 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zc772 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:53:09.411541 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:09.411130 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:53:09.942061 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:09.941980 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4vztp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:53:09.942061 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:09.942032 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:53:13.412156 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.412123 6234 generic.go:356] "Generic (PLEG): container finished" podID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerID="1c89b1a59bfd32c6c7f6d3f078a8282fa675c41a24d1197104e6a842cf12ab84" exitCode=0 Nov 03 22:53:13.412519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.412196 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" event={"ID":"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd","Type":"ContainerDied","Data":"1c89b1a59bfd32c6c7f6d3f078a8282fa675c41a24d1197104e6a842cf12ab84"} Nov 03 22:53:13.565749 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.565726 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:53:13.594805 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.594781 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:53:13.659378 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659352 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-kube-api-access-bq4cj\" (UniqueName: \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-kube-api-access-bq4cj\") pod \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " Nov 03 22:53:13.659504 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659405 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istiod-ca-cert\") pod \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " Nov 03 22:53:13.659504 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659444 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-socket\") pod \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " Nov 03 22:53:13.659504 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659467 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-token\" (UniqueName: \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-token\") pod \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " Nov 03 22:53:13.659504 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659487 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-credential-socket\") pod \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " Nov 03 22:53:13.659701 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659587 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-podinfo\") pod \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " Nov 03 22:53:13.659701 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659615 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-envoy\") pod \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " Nov 03 22:53:13.659812 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659708 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-certs\") pod \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " Nov 03 22:53:13.659812 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659752 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-data\") pod \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\" (UID: \"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd\") " Nov 03 22:53:13.659910 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659783 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-socket" pod "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" (UID: "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:13.659910 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659876 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-credential-socket" pod "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" (UID: "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:13.660045 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.660023 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-certs" pod "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" (UID: "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:13.660045 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.659884 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istiod-ca-cert" pod "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" (UID: "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:13.660149 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.660058 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-data" pod "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" (UID: "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:13.660225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.660207 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.660289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.660230 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.660289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.660244 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.660289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.660257 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.660289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.660272 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.661706 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.661681 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-podinfo" pod "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" (UID: "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:13.661816 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.661783 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-envoy" pod "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" (UID: "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:13.661870 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.661848 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-token" pod "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" (UID: "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:13.661913 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.661866 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-kube-api-access-bq4cj" pod "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" (UID: "017c1e6a-eb77-43f5-bdaf-e80ed469ddcd"). InnerVolumeSpecName "kube-api-access-bq4cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:13.761053 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761016 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-envoy\") pod \"4f58c897-de13-4ef8-ae65-7d343ec39dff\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " Nov 03 22:53:13.761221 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761063 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-credential-socket\") pod \"4f58c897-de13-4ef8-ae65-7d343ec39dff\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " Nov 03 22:53:13.761221 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761102 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-data\") pod \"4f58c897-de13-4ef8-ae65-7d343ec39dff\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " Nov 03 22:53:13.761221 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761140 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-socket\") pod \"4f58c897-de13-4ef8-ae65-7d343ec39dff\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " Nov 03 22:53:13.761221 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761172 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-token\" (UniqueName: \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-token\") pod \"4f58c897-de13-4ef8-ae65-7d343ec39dff\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " Nov 03 22:53:13.761457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761280 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-podinfo\") pod \"4f58c897-de13-4ef8-ae65-7d343ec39dff\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " Nov 03 22:53:13.761457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761322 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f58c897-de13-4ef8-ae65-7d343ec39dff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f58c897-de13-4ef8-ae65-7d343ec39dff-istiod-ca-cert\") pod \"4f58c897-de13-4ef8-ae65-7d343ec39dff\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " Nov 03 22:53:13.761457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761356 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-certs\") pod \"4f58c897-de13-4ef8-ae65-7d343ec39dff\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " Nov 03 22:53:13.761457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761383 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-kube-api-access-bgqfw\" (UniqueName: \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-kube-api-access-bgqfw\") pod \"4f58c897-de13-4ef8-ae65-7d343ec39dff\" (UID: \"4f58c897-de13-4ef8-ae65-7d343ec39dff\") " Nov 03 22:53:13.761457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761385 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-data" pod "4f58c897-de13-4ef8-ae65-7d343ec39dff" (UID: "4f58c897-de13-4ef8-ae65-7d343ec39dff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:13.761737 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761452 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-credential-socket" pod "4f58c897-de13-4ef8-ae65-7d343ec39dff" (UID: "4f58c897-de13-4ef8-ae65-7d343ec39dff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:13.761737 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761471 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-socket" pod "4f58c897-de13-4ef8-ae65-7d343ec39dff" (UID: "4f58c897-de13-4ef8-ae65-7d343ec39dff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:13.761737 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761695 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-certs" pod "4f58c897-de13-4ef8-ae65-7d343ec39dff" (UID: "4f58c897-de13-4ef8-ae65-7d343ec39dff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:13.761868 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761761 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f58c897-de13-4ef8-ae65-7d343ec39dff-istiod-ca-cert" pod "4f58c897-de13-4ef8-ae65-7d343ec39dff" (UID: "4f58c897-de13-4ef8-ae65-7d343ec39dff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:13.761868 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761842 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.761868 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761861 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.761977 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761871 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.761977 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761880 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.761977 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761889 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq4cj\" (UniqueName: \"kubernetes.io/projected/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd-kube-api-access-bq4cj\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.761977 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761900 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.761977 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761909 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.761977 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.761923 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.763733 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.763711 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-envoy" pod "4f58c897-de13-4ef8-ae65-7d343ec39dff" (UID: "4f58c897-de13-4ef8-ae65-7d343ec39dff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:13.763733 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.763720 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-podinfo" pod "4f58c897-de13-4ef8-ae65-7d343ec39dff" (UID: "4f58c897-de13-4ef8-ae65-7d343ec39dff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:13.763910 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.763891 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-kube-api-access-bgqfw" pod "4f58c897-de13-4ef8-ae65-7d343ec39dff" (UID: "4f58c897-de13-4ef8-ae65-7d343ec39dff"). InnerVolumeSpecName "kube-api-access-bgqfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:13.764025 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.764008 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-token" pod "4f58c897-de13-4ef8-ae65-7d343ec39dff" (UID: "4f58c897-de13-4ef8-ae65-7d343ec39dff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:13.862566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.862466 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.862566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.862518 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f58c897-de13-4ef8-ae65-7d343ec39dff-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.862566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.862532 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqfw\" (UniqueName: \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-kube-api-access-bgqfw\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.862566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.862543 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:13.862566 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:13.862552 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f58c897-de13-4ef8-ae65-7d343ec39dff-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:14.418520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.418466 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" Nov 03 22:53:14.418520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.418481 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp" event={"ID":"017c1e6a-eb77-43f5-bdaf-e80ed469ddcd","Type":"ContainerDied","Data":"00cd983792cd7c4212f3155219afd904b6a1a1bddd3fea639316f6b0af97838a"} Nov 03 22:53:14.418994 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.418570 6234 scope.go:122] "RemoveContainer" containerID="1c89b1a59bfd32c6c7f6d3f078a8282fa675c41a24d1197104e6a842cf12ab84" Nov 03 22:53:14.420000 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.419979 6234 generic.go:356] "Generic (PLEG): container finished" podID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerID="c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf" exitCode=0 Nov 03 22:53:14.420084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.420033 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" Nov 03 22:53:14.420084 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.420061 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" event={"ID":"4f58c897-de13-4ef8-ae65-7d343ec39dff","Type":"ContainerDied","Data":"c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf"} Nov 03 22:53:14.420192 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.420111 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772" event={"ID":"4f58c897-de13-4ef8-ae65-7d343ec39dff","Type":"ContainerDied","Data":"c2da560ad44a5c754a7e362c9b23e6bfcecacb66e20836dd2cfb3ac85fd38c6d"} Nov 03 22:53:14.428087 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.428065 6234 scope.go:122] "RemoveContainer" containerID="c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf" Nov 03 22:53:14.436770 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.436750 6234 scope.go:122] "RemoveContainer" containerID="c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf" Nov 03 22:53:14.437018 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:53:14.436997 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf\": container with ID starting with c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf not found: ID does not exist" containerID="c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf" Nov 03 22:53:14.437080 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.437027 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf"} err="failed to get container status \"c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf\": rpc error: code = NotFound desc = could not find container \"c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf\": container with ID starting with c528a22783c15b507d818a8cdbe3030a0914795fd28541b74f96ca76e115cdaf not found: ID does not exist" Nov 03 22:53:14.620545 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.620436 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772"] Nov 03 22:53:14.667254 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.667222 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zc772"] Nov 03 22:53:14.703291 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.703258 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" path="/var/lib/kubelet/pods/4f58c897-de13-4ef8-ae65-7d343ec39dff/volumes" Nov 03 22:53:14.725956 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.725928 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp"] Nov 03 22:53:14.730420 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:14.730398 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4vztp"] Nov 03 22:53:16.703806 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:16.703770 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" path="/var/lib/kubelet/pods/017c1e6a-eb77-43f5-bdaf-e80ed469ddcd/volumes" Nov 03 22:53:17.589716 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:17.589682 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn"] Nov 03 22:53:17.589922 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:17.589896 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" containerID="cri-o://2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1" gracePeriod=30 Nov 03 22:53:17.724939 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:17.724905 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:53:17.725307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:17.724960 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:53:22.355211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.355175 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:53:22.850618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.850589 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:53:22.950041 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950011 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-certs\") pod \"e7d62724-a2ac-43f0-8f35-9fc114109d79\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " Nov 03 22:53:22.950256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950061 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d62724-a2ac-43f0-8f35-9fc114109d79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d62724-a2ac-43f0-8f35-9fc114109d79-istiod-ca-cert\") pod \"e7d62724-a2ac-43f0-8f35-9fc114109d79\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " Nov 03 22:53:22.950256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950083 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-kube-api-access-q5xrg\" (UniqueName: \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-kube-api-access-q5xrg\") pod \"e7d62724-a2ac-43f0-8f35-9fc114109d79\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " Nov 03 22:53:22.950256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950109 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-socket\") pod \"e7d62724-a2ac-43f0-8f35-9fc114109d79\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " Nov 03 22:53:22.950256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950141 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-podinfo\") pod \"e7d62724-a2ac-43f0-8f35-9fc114109d79\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " Nov 03 22:53:22.950256 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950224 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-credential-socket\") pod \"e7d62724-a2ac-43f0-8f35-9fc114109d79\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " Nov 03 22:53:22.950588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950279 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-token\" (UniqueName: \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-token\") pod \"e7d62724-a2ac-43f0-8f35-9fc114109d79\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " Nov 03 22:53:22.950588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950313 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-data\") pod \"e7d62724-a2ac-43f0-8f35-9fc114109d79\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " Nov 03 22:53:22.950588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950341 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-envoy\") pod \"e7d62724-a2ac-43f0-8f35-9fc114109d79\" (UID: \"e7d62724-a2ac-43f0-8f35-9fc114109d79\") " Nov 03 22:53:22.950588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950318 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-certs" pod "e7d62724-a2ac-43f0-8f35-9fc114109d79" (UID: "e7d62724-a2ac-43f0-8f35-9fc114109d79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:22.950588 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950477 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d62724-a2ac-43f0-8f35-9fc114109d79-istiod-ca-cert" pod "e7d62724-a2ac-43f0-8f35-9fc114109d79" (UID: "e7d62724-a2ac-43f0-8f35-9fc114109d79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:22.950810 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950582 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-credential-socket" pod "e7d62724-a2ac-43f0-8f35-9fc114109d79" (UID: "e7d62724-a2ac-43f0-8f35-9fc114109d79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:22.950810 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950594 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-socket" pod "e7d62724-a2ac-43f0-8f35-9fc114109d79" (UID: "e7d62724-a2ac-43f0-8f35-9fc114109d79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:22.950810 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950719 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:22.950915 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.950848 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-data" pod "e7d62724-a2ac-43f0-8f35-9fc114109d79" (UID: "e7d62724-a2ac-43f0-8f35-9fc114109d79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:22.952530 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.952485 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-token" pod "e7d62724-a2ac-43f0-8f35-9fc114109d79" (UID: "e7d62724-a2ac-43f0-8f35-9fc114109d79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:22.952817 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.952795 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-kube-api-access-q5xrg" pod "e7d62724-a2ac-43f0-8f35-9fc114109d79" (UID: "e7d62724-a2ac-43f0-8f35-9fc114109d79"). InnerVolumeSpecName "kube-api-access-q5xrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:22.952904 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.952871 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-podinfo" pod "e7d62724-a2ac-43f0-8f35-9fc114109d79" (UID: "e7d62724-a2ac-43f0-8f35-9fc114109d79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:22.952963 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:22.952912 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-envoy" pod "e7d62724-a2ac-43f0-8f35-9fc114109d79" (UID: "e7d62724-a2ac-43f0-8f35-9fc114109d79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:23.051865 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.051830 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.051865 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.051858 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.051865 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.051867 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.051865 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.051876 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d62724-a2ac-43f0-8f35-9fc114109d79-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.052132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.051886 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5xrg\" (UniqueName: \"kubernetes.io/projected/e7d62724-a2ac-43f0-8f35-9fc114109d79-kube-api-access-q5xrg\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.052132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.051895 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.052132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.051903 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d62724-a2ac-43f0-8f35-9fc114109d79-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.052132 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.051911 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d62724-a2ac-43f0-8f35-9fc114109d79-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:23.475199 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.475161 6234 generic.go:356] "Generic (PLEG): container finished" podID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerID="2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1" exitCode=0 Nov 03 22:53:23.475686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.475219 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" Nov 03 22:53:23.475686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.475245 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" event={"ID":"e7d62724-a2ac-43f0-8f35-9fc114109d79","Type":"ContainerDied","Data":"2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1"} Nov 03 22:53:23.475686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.475299 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn" event={"ID":"e7d62724-a2ac-43f0-8f35-9fc114109d79","Type":"ContainerDied","Data":"252b4f38bc470262b3814ab7a232fd8fc2c9e7ebee087fb0c375bcb9f409b6e7"} Nov 03 22:53:23.475686 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.475321 6234 scope.go:122] "RemoveContainer" containerID="2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1" Nov 03 22:53:23.484851 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.484833 6234 scope.go:122] "RemoveContainer" containerID="bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d" Nov 03 22:53:23.493641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.493622 6234 scope.go:122] "RemoveContainer" containerID="2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1" Nov 03 22:53:23.493923 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:53:23.493905 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1\": container with ID starting with 2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1 not found: ID does not exist" containerID="2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1" Nov 03 22:53:23.493972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.493933 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1"} err="failed to get container status \"2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1\": rpc error: code = NotFound desc = could not find container \"2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1\": container with ID starting with 2aff61c46cf7abc4eca91830d58c2ba3e6c98e15a12dd0f623343ac70236b4d1 not found: ID does not exist" Nov 03 22:53:23.493972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.493952 6234 scope.go:122] "RemoveContainer" containerID="bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d" Nov 03 22:53:23.494181 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:53:23.494165 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d\": container with ID starting with bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d not found: ID does not exist" containerID="bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d" Nov 03 22:53:23.494225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.494186 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d"} err="failed to get container status \"bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d\": rpc error: code = NotFound desc = could not find container \"bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d\": container with ID starting with bc725341a0ab4f47aefbdb2d13ea3e688bb496b907b229574a04a005963e199d not found: ID does not exist" Nov 03 22:53:23.627705 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.627664 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn"] Nov 03 22:53:23.674483 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:23.674444 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-qxbxn"] Nov 03 22:53:24.703395 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:24.703357 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" path="/var/lib/kubelet/pods/e7d62724-a2ac-43f0-8f35-9fc114109d79/volumes" Nov 03 22:53:28.995704 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:53:28.995665 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:53:30.054748 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:30.054713 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj"] Nov 03 22:53:30.163419 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:30.163381 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk"] Nov 03 22:53:35.055996 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.055955 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" podUID="9ce525c2-9b52-4243-9b67-1d34d58aa1c9" containerName="payload-processing" containerID="cri-o://ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915" gracePeriod=25 Nov 03 22:53:35.164598 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.164533 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" podUID="6d154f05-576c-44a6-9301-9707cc2f629a" containerName="payload-pre-processing" containerID="cri-o://20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9" gracePeriod=25 Nov 03 22:53:35.300905 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.300881 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:53:35.370980 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.370948 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-plugins-config-volume\") pod \"9ce525c2-9b52-4243-9b67-1d34d58aa1c9\" (UID: \"9ce525c2-9b52-4243-9b67-1d34d58aa1c9\") " Nov 03 22:53:35.371151 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.371010 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-kube-api-access-mttv7\" (UniqueName: \"kubernetes.io/projected/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-kube-api-access-mttv7\") pod \"9ce525c2-9b52-4243-9b67-1d34d58aa1c9\" (UID: \"9ce525c2-9b52-4243-9b67-1d34d58aa1c9\") " Nov 03 22:53:35.371334 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.371313 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-plugins-config-volume" pod "9ce525c2-9b52-4243-9b67-1d34d58aa1c9" (UID: "9ce525c2-9b52-4243-9b67-1d34d58aa1c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:35.371448 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.371431 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:35.373352 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.373331 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-kube-api-access-mttv7" pod "9ce525c2-9b52-4243-9b67-1d34d58aa1c9" (UID: "9ce525c2-9b52-4243-9b67-1d34d58aa1c9"). InnerVolumeSpecName "kube-api-access-mttv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:35.400542 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.400518 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:53:35.472751 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.472722 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d154f05-576c-44a6-9301-9707cc2f629a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d154f05-576c-44a6-9301-9707cc2f629a-plugins-config-volume\") pod \"6d154f05-576c-44a6-9301-9707cc2f629a\" (UID: \"6d154f05-576c-44a6-9301-9707cc2f629a\") " Nov 03 22:53:35.472910 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.472790 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d154f05-576c-44a6-9301-9707cc2f629a-kube-api-access-9864w\" (UniqueName: \"kubernetes.io/projected/6d154f05-576c-44a6-9301-9707cc2f629a-kube-api-access-9864w\") pod \"6d154f05-576c-44a6-9301-9707cc2f629a\" (UID: \"6d154f05-576c-44a6-9301-9707cc2f629a\") " Nov 03 22:53:35.473430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.473387 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d154f05-576c-44a6-9301-9707cc2f629a-plugins-config-volume" pod "6d154f05-576c-44a6-9301-9707cc2f629a" (UID: "6d154f05-576c-44a6-9301-9707cc2f629a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:35.475549 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.473743 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mttv7\" (UniqueName: \"kubernetes.io/projected/9ce525c2-9b52-4243-9b67-1d34d58aa1c9-kube-api-access-mttv7\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:35.475549 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.473772 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d154f05-576c-44a6-9301-9707cc2f629a-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:35.480733 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.480704 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d154f05-576c-44a6-9301-9707cc2f629a-kube-api-access-9864w" pod "6d154f05-576c-44a6-9301-9707cc2f629a" (UID: "6d154f05-576c-44a6-9301-9707cc2f629a"). InnerVolumeSpecName "kube-api-access-9864w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:35.536428 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.536396 6234 generic.go:356] "Generic (PLEG): container finished" podID="9ce525c2-9b52-4243-9b67-1d34d58aa1c9" containerID="ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915" exitCode=0 Nov 03 22:53:35.536620 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.536456 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" Nov 03 22:53:35.536620 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.536481 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" event={"ID":"9ce525c2-9b52-4243-9b67-1d34d58aa1c9","Type":"ContainerDied","Data":"ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915"} Nov 03 22:53:35.536620 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.536593 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj" event={"ID":"9ce525c2-9b52-4243-9b67-1d34d58aa1c9","Type":"ContainerDied","Data":"ef6ff40052594a899f7eb52e0ebbdc158f9fa7ad4c76d4ccc9a0a2bff879d898"} Nov 03 22:53:35.536801 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.536625 6234 scope.go:122] "RemoveContainer" containerID="ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915" Nov 03 22:53:35.537983 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.537957 6234 generic.go:356] "Generic (PLEG): container finished" podID="6d154f05-576c-44a6-9301-9707cc2f629a" containerID="20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9" exitCode=0 Nov 03 22:53:35.538085 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.538010 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" Nov 03 22:53:35.538085 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.538045 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" event={"ID":"6d154f05-576c-44a6-9301-9707cc2f629a","Type":"ContainerDied","Data":"20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9"} Nov 03 22:53:35.538202 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.538096 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk" event={"ID":"6d154f05-576c-44a6-9301-9707cc2f629a","Type":"ContainerDied","Data":"4044bf8a04291595a12e2928caf158446532466cea430d63f962e6bcd264a401"} Nov 03 22:53:35.547677 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.547482 6234 scope.go:122] "RemoveContainer" containerID="ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915" Nov 03 22:53:35.547842 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:53:35.547819 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915\": container with ID starting with ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915 not found: ID does not exist" containerID="ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915" Nov 03 22:53:35.547959 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.547853 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915"} err="failed to get container status \"ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915\": rpc error: code = NotFound desc = could not find container \"ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915\": container with ID starting with ccdcf8a00dcfb6cf4d4a26fcc783fa476e80864d9458c5182ab497088c067915 not found: ID does not exist" Nov 03 22:53:35.547959 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.547871 6234 scope.go:122] "RemoveContainer" containerID="20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9" Nov 03 22:53:35.556917 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.556900 6234 scope.go:122] "RemoveContainer" containerID="20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9" Nov 03 22:53:35.557187 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:53:35.557165 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9\": container with ID starting with 20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9 not found: ID does not exist" containerID="20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9" Nov 03 22:53:35.557244 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.557194 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9"} err="failed to get container status \"20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9\": rpc error: code = NotFound desc = could not find container \"20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9\": container with ID starting with 20791cde76bcf1e5b7c30af9d7ed105aa74eb275a4d152c31b9b0a47fa85b6c9 not found: ID does not exist" Nov 03 22:53:35.574719 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.574698 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9864w\" (UniqueName: \"kubernetes.io/projected/6d154f05-576c-44a6-9301-9707cc2f629a-kube-api-access-9864w\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:35.688594 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.688556 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj"] Nov 03 22:53:35.692339 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.692310 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f81f8f-d676b878-g24rj"] Nov 03 22:53:35.794550 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.794514 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk"] Nov 03 22:53:35.797709 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:35.797680 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f81f8f-5647f5b5f9-bqkwk"] Nov 03 22:53:36.704238 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:36.704207 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d154f05-576c-44a6-9301-9707cc2f629a" path="/var/lib/kubelet/pods/6d154f05-576c-44a6-9301-9707cc2f629a/volumes" Nov 03 22:53:36.704641 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:36.704568 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce525c2-9b52-4243-9b67-1d34d58aa1c9" path="/var/lib/kubelet/pods/9ce525c2-9b52-4243-9b67-1d34d58aa1c9/volumes" Nov 03 22:53:49.866396 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:49.866360 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:53:50.859171 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:50.859134 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb"] Nov 03 22:53:50.859517 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:50.859446 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" podUID="43255448-b9e8-411d-8c9a-d3ca73526c1e" containerName="maas-api" containerID="cri-o://84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35" gracePeriod=30 Nov 03 22:53:51.126191 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.126167 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:53:51.232851 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.232741 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-maas-api-tls\") pod \"43255448-b9e8-411d-8c9a-d3ca73526c1e\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " Nov 03 22:53:51.233061 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.232914 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-metrics-certs\") pod \"43255448-b9e8-411d-8c9a-d3ca73526c1e\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " Nov 03 22:53:51.233061 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.233008 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43255448-b9e8-411d-8c9a-d3ca73526c1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43255448-b9e8-411d-8c9a-d3ca73526c1e-openshift-service-ca\") pod \"43255448-b9e8-411d-8c9a-d3ca73526c1e\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " Nov 03 22:53:51.233197 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.233089 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43255448-b9e8-411d-8c9a-d3ca73526c1e-kube-api-access-z8gds\" (UniqueName: \"kubernetes.io/projected/43255448-b9e8-411d-8c9a-d3ca73526c1e-kube-api-access-z8gds\") pod \"43255448-b9e8-411d-8c9a-d3ca73526c1e\" (UID: \"43255448-b9e8-411d-8c9a-d3ca73526c1e\") " Nov 03 22:53:51.233430 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.233392 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43255448-b9e8-411d-8c9a-d3ca73526c1e-openshift-service-ca" pod "43255448-b9e8-411d-8c9a-d3ca73526c1e" (UID: "43255448-b9e8-411d-8c9a-d3ca73526c1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:51.235484 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.235455 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-maas-api-tls" pod "43255448-b9e8-411d-8c9a-d3ca73526c1e" (UID: "43255448-b9e8-411d-8c9a-d3ca73526c1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:51.235815 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.235794 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-metrics-certs" pod "43255448-b9e8-411d-8c9a-d3ca73526c1e" (UID: "43255448-b9e8-411d-8c9a-d3ca73526c1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:51.235907 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.235799 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43255448-b9e8-411d-8c9a-d3ca73526c1e-kube-api-access-z8gds" pod "43255448-b9e8-411d-8c9a-d3ca73526c1e" (UID: "43255448-b9e8-411d-8c9a-d3ca73526c1e"). InnerVolumeSpecName "kube-api-access-z8gds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:51.334818 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.334787 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-metrics-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:51.334818 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.334811 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43255448-b9e8-411d-8c9a-d3ca73526c1e-openshift-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:51.334818 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.334822 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8gds\" (UniqueName: \"kubernetes.io/projected/43255448-b9e8-411d-8c9a-d3ca73526c1e-kube-api-access-z8gds\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:51.335041 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.334831 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43255448-b9e8-411d-8c9a-d3ca73526c1e-maas-api-tls\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:51.615693 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.615655 6234 generic.go:356] "Generic (PLEG): container finished" podID="43255448-b9e8-411d-8c9a-d3ca73526c1e" containerID="84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35" exitCode=0 Nov 03 22:53:51.615905 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.615712 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" Nov 03 22:53:51.615905 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.615738 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" event={"ID":"43255448-b9e8-411d-8c9a-d3ca73526c1e","Type":"ContainerDied","Data":"84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35"} Nov 03 22:53:51.615905 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.615794 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb" event={"ID":"43255448-b9e8-411d-8c9a-d3ca73526c1e","Type":"ContainerDied","Data":"6a1c2f57c220be9259ed3b4edcdfa887ef587bde7011d7a6801d4571d67165fb"} Nov 03 22:53:51.615905 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.615817 6234 scope.go:122] "RemoveContainer" containerID="84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35" Nov 03 22:53:51.625225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.625208 6234 scope.go:122] "RemoveContainer" containerID="84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35" Nov 03 22:53:51.625488 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:53:51.625471 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35\": container with ID starting with 84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35 not found: ID does not exist" containerID="84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35" Nov 03 22:53:51.625553 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.625512 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35"} err="failed to get container status \"84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35\": rpc error: code = NotFound desc = could not find container \"84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35\": container with ID starting with 84843f33bca71863c3039723b364981156733757842fe296b5d39617da5f2a35 not found: ID does not exist" Nov 03 22:53:51.768818 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.768785 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb"] Nov 03 22:53:51.771984 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:51.771962 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f81f8f-75dd588959-4jqtb"] Nov 03 22:53:52.703519 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:53:52.703456 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43255448-b9e8-411d-8c9a-d3ca73526c1e" path="/var/lib/kubelet/pods/43255448-b9e8-411d-8c9a-d3ca73526c1e/volumes" Nov 03 22:53:58.997006 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:53:58.996972 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:54:00.600805 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:00.599113 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc"] Nov 03 22:54:00.600805 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:00.600680 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" podUID="cae0e472-814e-405c-834c-1ab7a627652c" containerName="istio-proxy" containerID="cri-o://15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac" gracePeriod=30 Nov 03 22:54:05.851579 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.851554 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:54:05.873392 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873327 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-credential-socket\") pod \"cae0e472-814e-405c-834c-1ab7a627652c\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " Nov 03 22:54:05.873392 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873364 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-data\") pod \"cae0e472-814e-405c-834c-1ab7a627652c\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " Nov 03 22:54:05.873618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873403 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cae0e472-814e-405c-834c-1ab7a627652c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae0e472-814e-405c-834c-1ab7a627652c-istio-podinfo\") pod \"cae0e472-814e-405c-834c-1ab7a627652c\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " Nov 03 22:54:05.873618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873432 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-certs\") pod \"cae0e472-814e-405c-834c-1ab7a627652c\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " Nov 03 22:54:05.873618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873481 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-socket\") pod \"cae0e472-814e-405c-834c-1ab7a627652c\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " Nov 03 22:54:05.873618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873555 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-istio-token\" (UniqueName: \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-istio-token\") pod \"cae0e472-814e-405c-834c-1ab7a627652c\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " Nov 03 22:54:05.873618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873586 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-envoy\") pod \"cae0e472-814e-405c-834c-1ab7a627652c\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " Nov 03 22:54:05.873618 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873613 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae0e472-814e-405c-834c-1ab7a627652c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae0e472-814e-405c-834c-1ab7a627652c-istiod-ca-cert\") pod \"cae0e472-814e-405c-834c-1ab7a627652c\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " Nov 03 22:54:05.873943 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873648 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-kube-api-access-pbxzs\" (UniqueName: \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-kube-api-access-pbxzs\") pod \"cae0e472-814e-405c-834c-1ab7a627652c\" (UID: \"cae0e472-814e-405c-834c-1ab7a627652c\") " Nov 03 22:54:05.874006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873641 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-credential-socket" pod "cae0e472-814e-405c-834c-1ab7a627652c" (UID: "cae0e472-814e-405c-834c-1ab7a627652c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:05.874006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873740 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-data" pod "cae0e472-814e-405c-834c-1ab7a627652c" (UID: "cae0e472-814e-405c-834c-1ab7a627652c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:05.874006 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.873894 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-socket" pod "cae0e472-814e-405c-834c-1ab7a627652c" (UID: "cae0e472-814e-405c-834c-1ab7a627652c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:05.874173 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.874032 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae0e472-814e-405c-834c-1ab7a627652c-istiod-ca-cert" pod "cae0e472-814e-405c-834c-1ab7a627652c" (UID: "cae0e472-814e-405c-834c-1ab7a627652c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:05.874173 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.874117 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-certs" pod "cae0e472-814e-405c-834c-1ab7a627652c" (UID: "cae0e472-814e-405c-834c-1ab7a627652c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:05.875755 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.875733 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-kube-api-access-pbxzs" pod "cae0e472-814e-405c-834c-1ab7a627652c" (UID: "cae0e472-814e-405c-834c-1ab7a627652c"). InnerVolumeSpecName "kube-api-access-pbxzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:05.875909 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.875884 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-istio-token" pod "cae0e472-814e-405c-834c-1ab7a627652c" (UID: "cae0e472-814e-405c-834c-1ab7a627652c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:05.876093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.876071 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-envoy" pod "cae0e472-814e-405c-834c-1ab7a627652c" (UID: "cae0e472-814e-405c-834c-1ab7a627652c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:05.876093 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.876083 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cae0e472-814e-405c-834c-1ab7a627652c-istio-podinfo" pod "cae0e472-814e-405c-834c-1ab7a627652c" (UID: "cae0e472-814e-405c-834c-1ab7a627652c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:54:05.975211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.975181 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae0e472-814e-405c-834c-1ab7a627652c-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.975211 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.975209 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.975469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.975218 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.975469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.975228 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.975469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.975238 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.975469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.975246 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae0e472-814e-405c-834c-1ab7a627652c-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.975469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.975255 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbxzs\" (UniqueName: \"kubernetes.io/projected/cae0e472-814e-405c-834c-1ab7a627652c-kube-api-access-pbxzs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.975469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.975264 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:05.975469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:05.975272 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae0e472-814e-405c-834c-1ab7a627652c-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:06.688691 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:06.688658 6234 generic.go:356] "Generic (PLEG): container finished" podID="cae0e472-814e-405c-834c-1ab7a627652c" containerID="15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac" exitCode=0 Nov 03 22:54:06.688900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:06.688721 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" Nov 03 22:54:06.688900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:06.688748 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" event={"ID":"cae0e472-814e-405c-834c-1ab7a627652c","Type":"ContainerDied","Data":"15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac"} Nov 03 22:54:06.688900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:06.688802 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc" event={"ID":"cae0e472-814e-405c-834c-1ab7a627652c","Type":"ContainerDied","Data":"091fbe77dd2d34a314cf16d9c19b61a4217714896c16837cc556f4edcef97473"} Nov 03 22:54:06.688900 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:06.688823 6234 scope.go:122] "RemoveContainer" containerID="15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac" Nov 03 22:54:06.698307 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:06.698290 6234 scope.go:122] "RemoveContainer" containerID="15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac" Nov 03 22:54:06.698701 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:54:06.698680 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac\": container with ID starting with 15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac not found: ID does not exist" containerID="15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac" Nov 03 22:54:06.698781 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:06.698708 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac"} err="failed to get container status \"15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac\": rpc error: code = NotFound desc = could not find container \"15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac\": container with ID starting with 15bf93769e6a649496224563e941544de40e5190e5f12b7ff24aa3de128ae2ac not found: ID does not exist" Nov 03 22:54:06.888612 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:06.888582 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc"] Nov 03 22:54:06.935161 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:06.935133 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-k8zkc"] Nov 03 22:54:08.511861 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:08.511826 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:54:08.703274 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:08.703239 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae0e472-814e-405c-834c-1ab7a627652c" path="/var/lib/kubelet/pods/cae0e472-814e-405c-834c-1ab7a627652c/volumes" Nov 03 22:54:28.995341 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:54:28.995296 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:54:33.581546 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:54:33.581510 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:54:58.995543 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:54:58.995503 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:55:27.170812 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:27.170773 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:55:29.002254 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:55:29.002205 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:55:48.577406 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:48.577367 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:55:54.638709 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.638673 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb"] Nov 03 22:55:54.639107 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639054 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerName="istio-proxy" Nov 03 22:55:54.639107 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639066 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerName="istio-proxy" Nov 03 22:55:54.639107 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639082 6234 cpu_manager.go:414] "containerMap: removing container" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerName="istio-proxy" Nov 03 22:55:54.639107 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639088 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerName="istio-proxy" Nov 03 22:55:54.639107 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639097 6234 cpu_manager.go:414] "containerMap: removing container" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" Nov 03 22:55:54.639107 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639102 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639113 6234 cpu_manager.go:414] "containerMap: removing container" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639118 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639124 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639129 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639135 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639140 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639147 6234 cpu_manager.go:414] "containerMap: removing container" podUID="fd9fcdcb-09a9-4450-91a4-2c243670093a" containerName="payload-pre-processing" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639152 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9fcdcb-09a9-4450-91a4-2c243670093a" containerName="payload-pre-processing" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639160 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f" containerName="payload-processing" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639165 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f" containerName="payload-processing" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639171 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639175 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639185 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6d154f05-576c-44a6-9301-9707cc2f629a" containerName="payload-pre-processing" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639189 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d154f05-576c-44a6-9301-9707cc2f629a" containerName="payload-pre-processing" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639195 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639199 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639205 6234 cpu_manager.go:414] "containerMap: removing container" podUID="43255448-b9e8-411d-8c9a-d3ca73526c1e" containerName="maas-api" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639209 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43255448-b9e8-411d-8c9a-d3ca73526c1e" containerName="maas-api" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639215 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639219 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639228 6234 cpu_manager.go:414] "containerMap: removing container" podUID="cae0e472-814e-405c-834c-1ab7a627652c" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639232 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae0e472-814e-405c-834c-1ab7a627652c" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639239 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639243 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639250 6234 cpu_manager.go:414] "containerMap: removing container" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639254 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639261 6234 cpu_manager.go:414] "containerMap: removing container" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639266 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639274 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639278 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639284 6234 cpu_manager.go:414] "containerMap: removing container" podUID="7d7a8dea-f57e-40f0-8dfc-751843570fde" containerName="maas-api" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639288 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7a8dea-f57e-40f0-8dfc-751843570fde" containerName="maas-api" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639294 6234 cpu_manager.go:414] "containerMap: removing container" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639299 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639307 6234 cpu_manager.go:414] "containerMap: removing container" podUID="9ce525c2-9b52-4243-9b67-1d34d58aa1c9" containerName="payload-processing" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639312 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce525c2-9b52-4243-9b67-1d34d58aa1c9" containerName="payload-processing" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639317 6234 cpu_manager.go:414] "containerMap: removing container" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" Nov 03 22:55:54.639303 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639322 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639328 6234 cpu_manager.go:414] "containerMap: removing container" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639333 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639339 6234 cpu_manager.go:414] "containerMap: removing container" podUID="228e0c72-3449-49d8-a3c5-d96d0a8ccacc" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639344 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228e0c72-3449-49d8-a3c5-d96d0a8ccacc" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639348 6234 cpu_manager.go:414] "containerMap: removing container" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639353 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639360 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639364 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639371 6234 cpu_manager.go:414] "containerMap: removing container" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639375 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639384 6234 cpu_manager.go:414] "containerMap: removing container" podUID="735f20bd-8a1e-4c65-ac98-6bf603a6ecfa" containerName="payload-processing" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639388 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735f20bd-8a1e-4c65-ac98-6bf603a6ecfa" containerName="payload-processing" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639395 6234 cpu_manager.go:414] "containerMap: removing container" podUID="016058c4-df1f-4952-970d-e3f9ffe4704d" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639400 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016058c4-df1f-4952-970d-e3f9ffe4704d" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639406 6234 cpu_manager.go:414] "containerMap: removing container" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639411 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639417 6234 cpu_manager.go:414] "containerMap: removing container" podUID="2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" containerName="discovery" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639421 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" containerName="discovery" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639427 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639432 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639439 6234 cpu_manager.go:414] "containerMap: removing container" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639443 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639449 6234 cpu_manager.go:414] "containerMap: removing container" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639453 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639458 6234 cpu_manager.go:414] "containerMap: removing container" podUID="5163c5b4-4805-4adc-9216-8958b17a2b6d" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639462 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5163c5b4-4805-4adc-9216-8958b17a2b6d" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639469 6234 cpu_manager.go:414] "containerMap: removing container" podUID="baf62f19-94ba-4650-80a5-f19383286180" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639473 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf62f19-94ba-4650-80a5-f19383286180" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639479 6234 cpu_manager.go:414] "containerMap: removing container" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639483 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639505 6234 cpu_manager.go:414] "containerMap: removing container" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639510 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639563 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce525c2-9b52-4243-9b67-1d34d58aa1c9" containerName="payload-processing" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639570 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639577 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="494f6493-e301-4f79-af6b-db4768de6a77" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639583 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="228e0c72-3449-49d8-a3c5-d96d0a8ccacc" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639589 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="57101afc-e8db-439a-b8fe-ba005d025ea3" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639595 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639600 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f58c897-de13-4ef8-ae65-7d343ec39dff" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639621 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639627 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="735f20bd-8a1e-4c65-ac98-6bf603a6ecfa" containerName="payload-processing" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639634 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="016058c4-df1f-4952-970d-e3f9ffe4704d" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639640 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbbd364-d950-4467-85d3-fa4d0740adb8" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639647 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4b2a00-37a7-4c3b-ae3a-a901ae71815f" containerName="discovery" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639652 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fa3225-0006-485d-a7de-59dd408b6f99" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639658 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a3ff55-de90-431d-b281-3f2333cb7f1c" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639663 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="017c1e6a-eb77-43f5-bdaf-e80ed469ddcd" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639668 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="5163c5b4-4805-4adc-9216-8958b17a2b6d" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639675 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf62f19-94ba-4650-80a5-f19383286180" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639680 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb140cc5-3a6c-4b91-aadc-d803c46a9374" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639685 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca0821f-278d-456f-82e9-31578aa0b3a6" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639690 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10c1781-8bc9-4711-88b6-8b406d29ec25" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639696 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff3dd17-b8e0-4ce1-81d4-8ab4b72e1ee3" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639701 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b4fadd-1467-4cab-991b-08a56498eeae" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639707 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07f28fd-b732-4a25-ac79-2381cea9a252" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639712 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639717 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9fcdcb-09a9-4450-91a4-2c243670093a" containerName="payload-pre-processing" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639722 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f8c633-0135-48f9-b2f5-3d2a8cc06b3f" containerName="payload-processing" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639726 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24285a4-1ad4-4d6b-96da-75f575d24b5e" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639732 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d154f05-576c-44a6-9301-9707cc2f629a" containerName="payload-pre-processing" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639737 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3815cd-a626-41e9-9a6f-d5a271b218d5" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639741 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="43255448-b9e8-411d-8c9a-d3ca73526c1e" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639746 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6866d963-bd92-4452-be98-9bdd6a7e9e11" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639752 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae0e472-814e-405c-834c-1ab7a627652c" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639757 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a2bf96-b555-484f-b589-f3e68ab2878d" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639762 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb95e06-d3e2-4b7c-ad23-5bfbf2da70e1" containerName="istio-proxy" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639768 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6adb50-5a16-44e8-9f09-a45b32cafd1f" containerName="main" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639773 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7a8dea-f57e-40f0-8dfc-751843570fde" containerName="maas-api" Nov 03 22:55:54.640469 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.639778 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecb248e-7c95-4a4d-9892-57d24a6b1dc2" containerName="main" Nov 03 22:55:54.643027 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.643011 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.685776 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.685750 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665"] Nov 03 22:55:54.686183 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.686165 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" Nov 03 22:55:54.686282 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.686185 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e787c8-9530-4516-ac62-2de3e352fdab" containerName="istio-proxy" Nov 03 22:55:54.686341 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.686308 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ab3852-4c26-42b9-9927-926bb9a00e79" containerName="istio-proxy" Nov 03 22:55:54.686341 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.686324 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d62724-a2ac-43f0-8f35-9fc114109d79" containerName="istio-proxy" Nov 03 22:55:54.689358 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.689338 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.746063 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746028 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.746210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746072 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.746210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746099 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.746210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746169 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.746210 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746196 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.746378 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746220 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.746378 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746332 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.746378 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746355 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.746378 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746370 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.746573 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746485 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.746573 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746524 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.746573 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746548 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.746707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746615 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.746707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746638 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdrw2\" (UniqueName: \"kubernetes.io/projected/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-kube-api-access-hdrw2\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.746707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746653 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.746707 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746672 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87wfz\" (UniqueName: \"kubernetes.io/projected/7468570c-3547-4aa4-9d7e-c4f4aad4c395-kube-api-access-87wfz\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.746838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746716 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.746838 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.746769 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.833246 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.833210 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb"] Nov 03 22:55:54.847829 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.847797 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.848020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.847845 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.848020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.847883 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.848020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.847918 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.848020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.847949 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdrw2\" (UniqueName: \"kubernetes.io/projected/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-kube-api-access-hdrw2\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.848020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.847972 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.848020 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.847998 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87wfz\" (UniqueName: \"kubernetes.io/projected/7468570c-3547-4aa4-9d7e-c4f4aad4c395-kube-api-access-87wfz\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.848381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848023 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.848381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848064 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.848381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848136 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.848381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848174 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.848381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848200 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.848381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848236 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.848381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848277 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.848381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848305 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.848381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848339 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.848381 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848374 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.848964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848399 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.848964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848508 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.848964 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.848789 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.849151 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.849034 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.849217 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.849143 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.849457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.849391 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.849457 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.849427 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.849753 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.849691 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.849877 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.849857 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.850164 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.850138 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.850274 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.850255 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.850958 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.850932 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.851670 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.851649 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.851760 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.851744 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.852280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.852260 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.902687 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.902616 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7468570c-3547-4aa4-9d7e-c4f4aad4c395-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.903159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.903132 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdrw2\" (UniqueName: \"kubernetes.io/projected/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-kube-api-access-hdrw2\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.903609 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.903585 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e40f9f4-42b6-4c99-8b00-1ba515d7cf91-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-zh7jb\" (UID: \"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.903744 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.903724 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87wfz\" (UniqueName: \"kubernetes.io/projected/7468570c-3547-4aa4-9d7e-c4f4aad4c395-kube-api-access-87wfz\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-64665\" (UID: \"7468570c-3547-4aa4-9d7e-c4f4aad4c395\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:54.956225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.956191 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:54.969604 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:54.969573 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665"] Nov 03 22:55:55.002323 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:55.002291 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:55.378328 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:55.378292 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb"] Nov 03 22:55:55.384952 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:55:55.384922 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e40f9f4_42b6_4c99_8b00_1ba515d7cf91.slice/crio-22d523a1a72b76f88ca8bfd3ee278a99364680f361a00aa63df4c3751c148ea2 WatchSource:0}: Error finding container 22d523a1a72b76f88ca8bfd3ee278a99364680f361a00aa63df4c3751c148ea2: Status 404 returned error can't find the container with id 22d523a1a72b76f88ca8bfd3ee278a99364680f361a00aa63df4c3751c148ea2 Nov 03 22:55:55.387764 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:55.387733 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:55.387849 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:55.387798 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:55.387849 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:55.387827 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:55.432177 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:55.432154 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665"] Nov 03 22:55:55.439715 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:55:55.439691 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7468570c_3547_4aa4_9d7e_c4f4aad4c395.slice/crio-4f863eaf92db047966667590570a313356aac56116f119dd47b57c40e14a8f5a WatchSource:0}: Error finding container 4f863eaf92db047966667590570a313356aac56116f119dd47b57c40e14a8f5a: Status 404 returned error can't find the container with id 4f863eaf92db047966667590570a313356aac56116f119dd47b57c40e14a8f5a Nov 03 22:55:55.442460 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:55.442431 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:55.442561 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:55.442543 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:55.442614 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:55.442593 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:56.188713 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:56.188674 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" event={"ID":"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91","Type":"ContainerStarted","Data":"6356065630e0feedecbdce41877545209166431a49171c25369ae80660396d65"} Nov 03 22:55:56.190590 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:56.190560 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" event={"ID":"8e40f9f4-42b6-4c99-8b00-1ba515d7cf91","Type":"ContainerStarted","Data":"22d523a1a72b76f88ca8bfd3ee278a99364680f361a00aa63df4c3751c148ea2"} Nov 03 22:55:56.197224 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:56.197196 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" event={"ID":"7468570c-3547-4aa4-9d7e-c4f4aad4c395","Type":"ContainerStarted","Data":"ff432d15701e8640bcefcd836bf8bbf0cbe95bb73d0a239f5668d5af33d1143f"} Nov 03 22:55:56.200668 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:56.200635 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" event={"ID":"7468570c-3547-4aa4-9d7e-c4f4aad4c395","Type":"ContainerStarted","Data":"4f863eaf92db047966667590570a313356aac56116f119dd47b57c40e14a8f5a"} Nov 03 22:55:56.302159 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:56.302082 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" podStartSLOduration=2.302061993 podStartE2EDuration="2.302061993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:56.300533332 +0000 UTC m=+2875.712719349" watchObservedRunningTime="2026-11-03 22:55:56.302061993 +0000 UTC m=+2875.714248008" Nov 03 22:55:56.413484 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:56.413426 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" podStartSLOduration=2.413409673 podStartE2EDuration="2.413409673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:56.41206871 +0000 UTC m=+2875.824254729" watchObservedRunningTime="2026-11-03 22:55:56.413409673 +0000 UTC m=+2875.825595694" Nov 03 22:55:56.964520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:56.960676 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:55:56.964520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:56.962674 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:55:56.964520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:56.962738 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" podUID="8e40f9f4-42b6-4c99-8b00-1ba515d7cf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 22:55:57.008365 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:57.008321 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:55:57.008591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:57.008457 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-64665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:55:57.008591 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:57.008521 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" podUID="7468570c-3547-4aa4-9d7e-c4f4aad4c395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 22:55:57.957947 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:57.957905 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:55:57.958597 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:57.958566 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" podUID="8e40f9f4-42b6-4c99-8b00-1ba515d7cf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 22:55:58.003577 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:58.003550 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-64665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:55:58.003795 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:58.003775 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" podUID="7468570c-3547-4aa4-9d7e-c4f4aad4c395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 22:55:58.961540 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:58.961473 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:55:58.962294 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:58.962241 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" podUID="8e40f9f4-42b6-4c99-8b00-1ba515d7cf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 22:55:59.003114 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:59.003000 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-64665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:55:59.003114 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:59.003058 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" podUID="7468570c-3547-4aa4-9d7e-c4f4aad4c395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 22:55:59.011227 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:55:59.011195 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:55:59.958289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:59.957837 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:55:59.958289 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:55:59.957896 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" podUID="8e40f9f4-42b6-4c99-8b00-1ba515d7cf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 22:56:00.002927 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:00.002898 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-64665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:56:00.003382 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:00.003359 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" podUID="7468570c-3547-4aa4-9d7e-c4f4aad4c395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 22:56:00.715623 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:00.715593 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:56:00.964518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:00.962887 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:56:00.964518 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:00.962948 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" podUID="8e40f9f4-42b6-4c99-8b00-1ba515d7cf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 22:56:01.003297 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:01.003210 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-64665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:56:01.003297 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:01.003260 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" podUID="7468570c-3547-4aa4-9d7e-c4f4aad4c395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 22:56:01.956984 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:01.956925 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:56:01.957164 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:01.957008 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" podUID="8e40f9f4-42b6-4c99-8b00-1ba515d7cf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 22:56:02.003526 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:02.003473 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-64665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:56:02.003930 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:02.003554 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" podUID="7468570c-3547-4aa4-9d7e-c4f4aad4c395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 22:56:02.957568 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:02.957527 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:56:02.957748 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:02.957594 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" podUID="8e40f9f4-42b6-4c99-8b00-1ba515d7cf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 22:56:03.003178 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:03.003121 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-64665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 22:56:03.003402 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:03.003204 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" podUID="7468570c-3547-4aa4-9d7e-c4f4aad4c395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 22:56:03.960718 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:03.960688 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:56:03.961118 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:03.961054 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:56:03.961869 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:03.961843 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" Nov 03 22:56:04.006284 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:04.006250 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:56:04.006669 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:04.006650 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:56:04.007225 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:04.007208 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" Nov 03 22:56:09.746438 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.746398 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m"] Nov 03 22:56:09.750036 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.750016 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.835814 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.835740 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m"] Nov 03 22:56:09.887931 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.887895 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.887931 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.887934 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.888137 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.888038 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.888137 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.888073 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.888137 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.888093 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.888277 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.888251 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.888386 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.888363 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3294b5-bf10-41a8-b126-6393399d4b40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.888522 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.888486 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.888597 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.888578 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfs4n\" (UniqueName: \"kubernetes.io/projected/7f3294b5-bf10-41a8-b126-6393399d4b40-kube-api-access-lfs4n\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990154 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990119 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990154 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990159 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990379 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990266 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990379 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990298 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990379 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990339 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3294b5-bf10-41a8-b126-6393399d4b40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990400 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990447 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfs4n\" (UniqueName: \"kubernetes.io/projected/7f3294b5-bf10-41a8-b126-6393399d4b40-kube-api-access-lfs4n\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990516 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990484 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990675 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990541 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990858 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990836 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.990957 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.990893 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.991037 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.991003 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.991250 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.991225 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.991374 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.991278 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3294b5-bf10-41a8-b126-6393399d4b40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.992569 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.992549 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:09.992793 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:09.992777 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:10.045230 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:10.045165 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3294b5-bf10-41a8-b126-6393399d4b40-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:10.045342 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:10.045271 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfs4n\" (UniqueName: \"kubernetes.io/projected/7f3294b5-bf10-41a8-b126-6393399d4b40-kube-api-access-lfs4n\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-w4b5m\" (UID: \"7f3294b5-bf10-41a8-b126-6393399d4b40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:10.062650 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:10.062632 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:10.480215 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:10.480175 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m"] Nov 03 22:56:10.486408 ip-10-0-134-57 kubenswrapper[6234]: W1103 22:56:10.486377 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3294b5_bf10_41a8_b126_6393399d4b40.slice/crio-7fc368b86287943ccc5712e5efd865963aa5014717a9eb95bfcbfef8b98e8292 WatchSource:0}: Error finding container 7fc368b86287943ccc5712e5efd865963aa5014717a9eb95bfcbfef8b98e8292: Status 404 returned error can't find the container with id 7fc368b86287943ccc5712e5efd865963aa5014717a9eb95bfcbfef8b98e8292 Nov 03 22:56:10.488717 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:10.488683 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:56:10.488818 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:10.488751 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:56:10.488818 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:10.488778 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:56:11.280770 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:11.280734 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" event={"ID":"7f3294b5-bf10-41a8-b126-6393399d4b40","Type":"ContainerStarted","Data":"56b6b8d10e2c936b497bfb516d9b2d2f0e8f6f93f97a6d561b263324f2fc2829"} Nov 03 22:56:11.281201 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:11.280786 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" event={"ID":"7f3294b5-bf10-41a8-b126-6393399d4b40","Type":"ContainerStarted","Data":"7fc368b86287943ccc5712e5efd865963aa5014717a9eb95bfcbfef8b98e8292"} Nov 03 22:56:11.436355 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:11.436306 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" podStartSLOduration=2.43629404 podStartE2EDuration="2.43629404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:11.435784042 +0000 UTC m=+2890.847970060" watchObservedRunningTime="2026-11-03 22:56:11.43629404 +0000 UTC m=+2890.848480056" Nov 03 22:56:12.063991 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:12.063959 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:12.065702 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:12.065674 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:56:12.065862 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:12.065721 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" podUID="7f3294b5-bf10-41a8-b126-6393399d4b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 22:56:13.064009 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:13.063966 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:56:13.064451 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:13.064024 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" podUID="7f3294b5-bf10-41a8-b126-6393399d4b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 22:56:14.063968 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:14.063935 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:56:14.064183 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:14.063989 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" podUID="7f3294b5-bf10-41a8-b126-6393399d4b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 22:56:15.063942 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:15.063897 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:56:15.064125 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:15.063992 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" podUID="7f3294b5-bf10-41a8-b126-6393399d4b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 22:56:16.063972 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:16.063941 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:56:16.064402 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:16.063993 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" podUID="7f3294b5-bf10-41a8-b126-6393399d4b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 22:56:17.063688 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:17.063652 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:56:17.063866 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:17.063706 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" podUID="7f3294b5-bf10-41a8-b126-6393399d4b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 22:56:18.673113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:18.673078 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:18.673672 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:18.673644 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:18.674422 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:18.674403 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" Nov 03 22:56:20.098575 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:20.098543 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:56:28.996097 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:56:28.996059 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:56:38.795655 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:56:38.795619 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:56:58.995167 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:56:58.995125 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:57:11.070602 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:57:11.070569 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:57:21.796198 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:57:21.796166 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:57:29.013987 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:57:29.013947 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:57:34.061113 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:57:34.061077 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:57:56.857233 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:57:56.857119 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:57:58.996421 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:57:58.996386 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:58:04.325520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:58:04.324588 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:58:04.325520 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:58:04.324651 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" podUID="fce20530-f3fd-4d1a-9dcd-151248db648a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:58:06.073845 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:58:06.073808 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:58:25.066925 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:58:25.066887 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:58:28.995611 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:58:28.995572 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:58:34.328632 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:58:34.328589 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:58:34.329695 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:58:34.329227 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" podUID="fce20530-f3fd-4d1a-9dcd-151248db648a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:58:37.310280 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:58:37.310236 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:58:57.490509 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:58:57.490464 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:58:59.099970 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:58:59.099770 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:59:16.479667 ip-10-0-134-57 kubenswrapper[6234]: I1103 22:59:16.479582 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 22:59:28.996437 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:59:28.996396 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 22:59:59.005109 ip-10-0-134-57 kubenswrapper[6234]: E1103 22:59:59.005070 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:00:00.596700 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:00.596667 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:00:23.893851 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:23.893813 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:00:28.996324 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:00:28.996287 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:00:34.772434 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:34.772402 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:00:55.682940 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.682901 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6"] Nov 03 23:00:55.686713 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.686687 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.748924 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.748893 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-fe3af2-openshift-default-dockercfg-bjsnh\"" Nov 03 23:00:55.783965 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.783930 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6"] Nov 03 23:00:55.828431 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.828400 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istiod-ca-cert\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.828625 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.828443 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-data\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.828625 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.828468 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-credential-socket\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.828734 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.828625 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-podinfo\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.828734 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.828661 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-certs\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.828734 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.828681 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-envoy\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.828872 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.828769 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxtp\" (UniqueName: \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-kube-api-access-6xxtp\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.828872 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.828805 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-socket\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.828872 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.828827 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-token\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.930237 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.930200 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-data\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.930237 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.930248 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-credential-socket\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.930760 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.930298 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-podinfo\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.930760 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.930327 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-certs\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.930760 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.930379 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-envoy\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.930760 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.930464 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxtp\" (UniqueName: \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-kube-api-access-6xxtp\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.930760 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.930531 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-socket\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.930760 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.930570 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-token\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.930760 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.930670 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-data\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.930760 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.930675 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istiod-ca-cert\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.931344 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.931019 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-socket\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.931344 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.931297 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-credential-socket\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.931344 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.931326 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istiod-ca-cert\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.933054 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.931734 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-certs\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.934291 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.934260 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-envoy\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.934407 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.934391 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-podinfo\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.985845 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.985814 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-token\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.986113 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.986090 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxtp\" (UniqueName: \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-kube-api-access-6xxtp\") pod \"e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:55.998964 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:55.998938 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:56.416649 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:56.416568 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6"] Nov 03 23:00:56.423636 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:00:56.423607 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78ced59_6b45_4dc3_8adf_a7bc342bba0e.slice/crio-19f8fe83ebd46bc13a6610394440f6b3bb0171e87b9449ada7f7d30941407ba6 WatchSource:0}: Error finding container 19f8fe83ebd46bc13a6610394440f6b3bb0171e87b9449ada7f7d30941407ba6: Status 404 returned error can't find the container with id 19f8fe83ebd46bc13a6610394440f6b3bb0171e87b9449ada7f7d30941407ba6 Nov 03 23:00:56.425797 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:56.425763 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:00:56.425910 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:56.425845 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:00:56.425910 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:56.425890 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:00:56.871542 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:56.871480 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" event={"ID":"d78ced59-6b45-4dc3-8adf-a7bc342bba0e","Type":"ContainerStarted","Data":"ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7"} Nov 03 23:00:56.871970 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:56.871558 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" event={"ID":"d78ced59-6b45-4dc3-8adf-a7bc342bba0e","Type":"ContainerStarted","Data":"19f8fe83ebd46bc13a6610394440f6b3bb0171e87b9449ada7f7d30941407ba6"} Nov 03 23:00:56.981578 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:56.981534 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" podStartSLOduration=1.981518729 podStartE2EDuration="1.981518729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:56.981448284 +0000 UTC m=+3176.393634312" watchObservedRunningTime="2026-11-03 23:00:56.981518729 +0000 UTC m=+3176.393704742" Nov 03 23:00:56.999838 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:56.999812 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:58.004562 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:58.004534 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:58.881651 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:58.881622 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:58.882750 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:00:58.882726 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:00:58.998549 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:00:58.998489 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:01:03.625596 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.625563 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl"] Nov 03 23:01:03.629158 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.629139 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:01:03.703940 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.703907 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dl6\" (UniqueName: \"kubernetes.io/projected/a4a542be-eda5-4ccb-8669-637023d7f739-kube-api-access-42dl6\") pod \"payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl\" (UID: \"a4a542be-eda5-4ccb-8669-637023d7f739\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:01:03.704096 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.704038 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a542be-eda5-4ccb-8669-637023d7f739-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl\" (UID: \"a4a542be-eda5-4ccb-8669-637023d7f739\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:01:03.727965 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.727928 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-fe3af2\"" Nov 03 23:01:03.728283 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.727928 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-2f7d23e4-dockercfg-shdxw\"" Nov 03 23:01:03.729819 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.729792 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl"] Nov 03 23:01:03.805168 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.805132 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dl6\" (UniqueName: \"kubernetes.io/projected/a4a542be-eda5-4ccb-8669-637023d7f739-kube-api-access-42dl6\") pod \"payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl\" (UID: \"a4a542be-eda5-4ccb-8669-637023d7f739\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:01:03.805344 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.805221 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a542be-eda5-4ccb-8669-637023d7f739-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl\" (UID: \"a4a542be-eda5-4ccb-8669-637023d7f739\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:01:03.805892 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.805872 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a542be-eda5-4ccb-8669-637023d7f739-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl\" (UID: \"a4a542be-eda5-4ccb-8669-637023d7f739\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:01:03.858882 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.858853 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dl6\" (UniqueName: \"kubernetes.io/projected/a4a542be-eda5-4ccb-8669-637023d7f739-kube-api-access-42dl6\") pod \"payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl\" (UID: \"a4a542be-eda5-4ccb-8669-637023d7f739\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:01:03.942970 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:03.942920 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:01:04.309529 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:04.309431 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl"] Nov 03 23:01:04.315507 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:01:04.315468 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a542be_eda5_4ccb_8669_637023d7f739.slice/crio-b5160c14d2fa6658497e1edab49b78f448a87fb4bdeec9e01647811a8aa56512 WatchSource:0}: Error finding container b5160c14d2fa6658497e1edab49b78f448a87fb4bdeec9e01647811a8aa56512: Status 404 returned error can't find the container with id b5160c14d2fa6658497e1edab49b78f448a87fb4bdeec9e01647811a8aa56512 Nov 03 23:01:04.909920 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:04.909884 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" event={"ID":"a4a542be-eda5-4ccb-8669-637023d7f739","Type":"ContainerStarted","Data":"8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1"} Nov 03 23:01:04.910322 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:04.909937 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" event={"ID":"a4a542be-eda5-4ccb-8669-637023d7f739","Type":"ContainerStarted","Data":"b5160c14d2fa6658497e1edab49b78f448a87fb4bdeec9e01647811a8aa56512"} Nov 03 23:01:04.910322 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:04.909989 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:01:05.017405 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:05.017346 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" podStartSLOduration=2.017333787 podStartE2EDuration="2.017333787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:05.016417909 +0000 UTC m=+3184.428603927" watchObservedRunningTime="2026-11-03 23:01:05.017333787 +0000 UTC m=+3184.429519803" Nov 03 23:01:14.169055 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:14.169010 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:01:15.918610 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:15.918582 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:01:24.016797 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:24.016760 6234 generic.go:356] "Generic (PLEG): container finished" podID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerID="ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7" exitCode=0 Nov 03 23:01:24.017351 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:24.016802 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" event={"ID":"d78ced59-6b45-4dc3-8adf-a7bc342bba0e","Type":"ContainerDied","Data":"ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7"} Nov 03 23:01:24.017351 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:24.017313 6234 scope.go:122] "RemoveContainer" containerID="ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7" Nov 03 23:01:24.018585 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:24.018553 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:01:24.018678 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:24.018623 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:01:24.018678 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:24.018655 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:01:25.023097 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:25.023064 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" event={"ID":"d78ced59-6b45-4dc3-8adf-a7bc342bba0e","Type":"ContainerStarted","Data":"59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a"} Nov 03 23:01:25.999368 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:25.999337 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:01:26.000565 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:26.000540 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 23:01:26.000704 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:26.000589 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 23:01:26.999512 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:26.999470 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 23:01:27.000007 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:26.999544 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 23:01:27.999688 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:27.999654 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 23:01:28.000125 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:27.999715 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 23:01:28.847522 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.845314 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp"] Nov 03 23:01:28.868728 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.868697 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:28.882374 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.882347 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:01:28.899469 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.899436 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgdp\" (UniqueName: \"kubernetes.io/projected/8156999a-b200-4d93-b27b-11856682a544-kube-api-access-hcgdp\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:28.899720 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.899690 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8156999a-b200-4d93-b27b-11856682a544-tls-certs\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:28.899720 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.899729 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-dshm\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:28.899935 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.899754 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-home\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:28.900156 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.900131 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-model-cache\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:28.900235 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.900195 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-tmp-dir\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:28.912309 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.912282 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp"] Nov 03 23:01:28.963518 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.962939 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fe3af2\"/\"openshift-service-ca.crt\"" Nov 03 23:01:28.963518 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.963096 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fe3af2\"/\"default-dockercfg-wv4nb\"" Nov 03 23:01:28.964724 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.964691 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fe3af2\"/\"simulator-single-main-fe3af2-kserve-self-signed-certs\"" Nov 03 23:01:28.964851 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:28.964726 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fe3af2\"/\"kube-root-ca.crt\"" Nov 03 23:01:29.001348 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.001312 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgdp\" (UniqueName: \"kubernetes.io/projected/8156999a-b200-4d93-b27b-11856682a544-kube-api-access-hcgdp\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.002076 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.001404 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8156999a-b200-4d93-b27b-11856682a544-tls-certs\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.002076 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.001451 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-dshm\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.002076 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.001528 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-home\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.002076 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.001563 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-model-cache\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.002076 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.001641 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-tmp-dir\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.005214 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.005158 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 23:01:29.005378 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.005250 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 23:01:29.005982 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.005952 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-home\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.007007 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.006969 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-model-cache\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.007818 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.007768 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8156999a-b200-4d93-b27b-11856682a544-tls-certs\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.008569 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.008389 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-tmp-dir\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.010076 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:01:29.010050 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:01:29.012815 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.012790 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-dshm\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.066430 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.066401 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgdp\" (UniqueName: \"kubernetes.io/projected/8156999a-b200-4d93-b27b-11856682a544-kube-api-access-hcgdp\") pod \"simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.180370 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.180277 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:29.555450 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:29.555360 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp"] Nov 03 23:01:30.067709 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:30.067540 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:01:30.069703 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:30.069670 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:01:30.075342 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:30.075311 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" event={"ID":"8156999a-b200-4d93-b27b-11856682a544","Type":"ContainerStarted","Data":"0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81"} Nov 03 23:01:30.075473 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:30.075354 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" event={"ID":"8156999a-b200-4d93-b27b-11856682a544","Type":"ContainerStarted","Data":"76a35df106e1c882261c7ab20e67db705427d8d0c41e79a3d6d21b84637d8d7a"} Nov 03 23:01:30.181224 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:30.181197 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:30.439032 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:30.438981 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" podStartSLOduration=2.438965624 podStartE2EDuration="2.438965624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:30.438245303 +0000 UTC m=+3209.850431319" watchObservedRunningTime="2026-11-03 23:01:30.438965624 +0000 UTC m=+3209.851151641" Nov 03 23:01:33.404471 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:33.404433 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:01:39.181544 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:39.181487 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:39.194154 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:39.194127 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:40.134471 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:40.134439 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:01:58.562563 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:01:58.562520 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:01:59.012368 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:01:59.012320 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:02:27.606692 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:27.606612 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:02:28.996575 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:02:28.996531 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:02:29.492415 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:29.492374 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp"] Nov 03 23:02:36.241667 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:36.241630 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl"] Nov 03 23:02:39.449351 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:39.449307 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:02:41.242373 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:41.242328 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" podUID="a4a542be-eda5-4ccb-8669-637023d7f739" containerName="payload-pre-processing" containerID="cri-o://8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1" gracePeriod=25 Nov 03 23:02:44.513444 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.513384 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" podUID="8156999a-b200-4d93-b27b-11856682a544" containerName="main" containerID="cri-o://0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81" gracePeriod=45 Nov 03 23:02:44.762753 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.762723 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:02:44.775927 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.775860 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8156999a-b200-4d93-b27b-11856682a544-tls-certs\" (UniqueName: \"kubernetes.io/secret/8156999a-b200-4d93-b27b-11856682a544-tls-certs\") pod \"8156999a-b200-4d93-b27b-11856682a544\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " Nov 03 23:02:44.775927 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.775911 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8156999a-b200-4d93-b27b-11856682a544-kube-api-access-hcgdp\" (UniqueName: \"kubernetes.io/projected/8156999a-b200-4d93-b27b-11856682a544-kube-api-access-hcgdp\") pod \"8156999a-b200-4d93-b27b-11856682a544\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " Nov 03 23:02:44.776126 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.775935 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-home\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-home\") pod \"8156999a-b200-4d93-b27b-11856682a544\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " Nov 03 23:02:44.776126 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.775971 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-dshm\") pod \"8156999a-b200-4d93-b27b-11856682a544\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " Nov 03 23:02:44.776126 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.776005 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-tmp-dir\") pod \"8156999a-b200-4d93-b27b-11856682a544\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " Nov 03 23:02:44.776126 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.776109 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-model-cache\") pod \"8156999a-b200-4d93-b27b-11856682a544\" (UID: \"8156999a-b200-4d93-b27b-11856682a544\") " Nov 03 23:02:44.776568 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.776179 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-home" pod "8156999a-b200-4d93-b27b-11856682a544" (UID: "8156999a-b200-4d93-b27b-11856682a544"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:44.776568 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.776381 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-model-cache" pod "8156999a-b200-4d93-b27b-11856682a544" (UID: "8156999a-b200-4d93-b27b-11856682a544"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:44.776920 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.776681 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:44.776920 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.776900 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:44.777023 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.776976 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-tmp-dir" pod "8156999a-b200-4d93-b27b-11856682a544" (UID: "8156999a-b200-4d93-b27b-11856682a544"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:44.778291 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.778263 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-dshm" pod "8156999a-b200-4d93-b27b-11856682a544" (UID: "8156999a-b200-4d93-b27b-11856682a544"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:44.778716 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.778690 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8156999a-b200-4d93-b27b-11856682a544-tls-certs" pod "8156999a-b200-4d93-b27b-11856682a544" (UID: "8156999a-b200-4d93-b27b-11856682a544"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:44.778814 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.778714 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8156999a-b200-4d93-b27b-11856682a544-kube-api-access-hcgdp" pod "8156999a-b200-4d93-b27b-11856682a544" (UID: "8156999a-b200-4d93-b27b-11856682a544"). InnerVolumeSpecName "kube-api-access-hcgdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:44.878470 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.878434 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8156999a-b200-4d93-b27b-11856682a544-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:44.878470 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.878465 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgdp\" (UniqueName: \"kubernetes.io/projected/8156999a-b200-4d93-b27b-11856682a544-kube-api-access-hcgdp\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:44.878470 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.878477 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:44.878804 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:44.878487 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8156999a-b200-4d93-b27b-11856682a544-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:45.433840 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:45.433808 6234 generic.go:356] "Generic (PLEG): container finished" podID="8156999a-b200-4d93-b27b-11856682a544" containerID="0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81" exitCode=0 Nov 03 23:02:45.434029 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:45.433848 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" Nov 03 23:02:45.434029 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:45.433895 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" event={"ID":"8156999a-b200-4d93-b27b-11856682a544","Type":"ContainerDied","Data":"0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81"} Nov 03 23:02:45.434029 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:45.433936 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp" event={"ID":"8156999a-b200-4d93-b27b-11856682a544","Type":"ContainerDied","Data":"76a35df106e1c882261c7ab20e67db705427d8d0c41e79a3d6d21b84637d8d7a"} Nov 03 23:02:45.434029 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:45.433957 6234 scope.go:122] "RemoveContainer" containerID="0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81" Nov 03 23:02:45.443192 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:45.443173 6234 scope.go:122] "RemoveContainer" containerID="0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81" Nov 03 23:02:45.443443 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:02:45.443424 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81\": container with ID starting with 0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81 not found: ID does not exist" containerID="0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81" Nov 03 23:02:45.443569 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:45.443452 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81"} err="failed to get container status \"0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81\": rpc error: code = NotFound desc = could not find container \"0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81\": container with ID starting with 0715cab83531a00c3855a12715c2c2ddeca2a6ca4bbccfbdad2b58a18e503d81 not found: ID does not exist" Nov 03 23:02:45.587090 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:45.587059 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp"] Nov 03 23:02:45.633772 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:45.633736 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-fe3af2/simulator-single-main-fe3af2-kserve-77bfd9fb6c-kkkrp"] Nov 03 23:02:45.916896 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:45.916811 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" podUID="a4a542be-eda5-4ccb-8669-637023d7f739" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:02:46.703804 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:46.703759 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8156999a-b200-4d93-b27b-11856682a544" path="/var/lib/kubelet/pods/8156999a-b200-4d93-b27b-11856682a544/volumes" Nov 03 23:02:47.902291 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:47.902257 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6"] Nov 03 23:02:47.902807 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:47.902489 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" containerID="cri-o://59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a" gracePeriod=30 Nov 03 23:02:53.150597 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.150571 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:02:53.272684 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.272652 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-data\") pod \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " Nov 03 23:02:53.272869 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.272707 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-token\" (UniqueName: \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-token\") pod \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " Nov 03 23:02:53.272869 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.272736 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-certs\") pod \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " Nov 03 23:02:53.272869 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.272810 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-credential-socket\") pod \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " Nov 03 23:02:53.272869 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.272839 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-socket\") pod \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " Nov 03 23:02:53.273109 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.272888 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-podinfo\") pod \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " Nov 03 23:02:53.273109 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.272898 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-data" pod "d78ced59-6b45-4dc3-8adf-a7bc342bba0e" (UID: "d78ced59-6b45-4dc3-8adf-a7bc342bba0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:53.273109 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.272941 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istiod-ca-cert\") pod \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " Nov 03 23:02:53.273109 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.272975 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-kube-api-access-6xxtp\" (UniqueName: \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-kube-api-access-6xxtp\") pod \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " Nov 03 23:02:53.273109 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.273037 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-envoy\") pod \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\" (UID: \"d78ced59-6b45-4dc3-8adf-a7bc342bba0e\") " Nov 03 23:02:53.273371 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.273137 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-credential-socket" pod "d78ced59-6b45-4dc3-8adf-a7bc342bba0e" (UID: "d78ced59-6b45-4dc3-8adf-a7bc342bba0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:53.273371 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.273149 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-certs" pod "d78ced59-6b45-4dc3-8adf-a7bc342bba0e" (UID: "d78ced59-6b45-4dc3-8adf-a7bc342bba0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:53.273371 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.273165 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-socket" pod "d78ced59-6b45-4dc3-8adf-a7bc342bba0e" (UID: "d78ced59-6b45-4dc3-8adf-a7bc342bba0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:53.273556 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.273431 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istiod-ca-cert" pod "d78ced59-6b45-4dc3-8adf-a7bc342bba0e" (UID: "d78ced59-6b45-4dc3-8adf-a7bc342bba0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:53.273881 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.273852 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.274036 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.273885 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.274036 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.273902 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.274036 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.273917 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.274036 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.273937 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.275301 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.275276 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-kube-api-access-6xxtp" pod "d78ced59-6b45-4dc3-8adf-a7bc342bba0e" (UID: "d78ced59-6b45-4dc3-8adf-a7bc342bba0e"). InnerVolumeSpecName "kube-api-access-6xxtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:53.275407 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.275326 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-token" pod "d78ced59-6b45-4dc3-8adf-a7bc342bba0e" (UID: "d78ced59-6b45-4dc3-8adf-a7bc342bba0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:53.275615 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.275589 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-podinfo" pod "d78ced59-6b45-4dc3-8adf-a7bc342bba0e" (UID: "d78ced59-6b45-4dc3-8adf-a7bc342bba0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:02:53.275751 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.275728 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-envoy" pod "d78ced59-6b45-4dc3-8adf-a7bc342bba0e" (UID: "d78ced59-6b45-4dc3-8adf-a7bc342bba0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:53.374915 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.374884 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.374915 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.374912 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.375122 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.374931 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xxtp\" (UniqueName: \"kubernetes.io/projected/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-kube-api-access-6xxtp\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.375122 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.374940 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78ced59-6b45-4dc3-8adf-a7bc342bba0e-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:53.473777 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.473739 6234 generic.go:356] "Generic (PLEG): container finished" podID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerID="59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a" exitCode=0 Nov 03 23:02:53.473944 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.473789 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" event={"ID":"d78ced59-6b45-4dc3-8adf-a7bc342bba0e","Type":"ContainerDied","Data":"59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a"} Nov 03 23:02:53.473944 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.473795 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" Nov 03 23:02:53.473944 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.473843 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6" event={"ID":"d78ced59-6b45-4dc3-8adf-a7bc342bba0e","Type":"ContainerDied","Data":"19f8fe83ebd46bc13a6610394440f6b3bb0171e87b9449ada7f7d30941407ba6"} Nov 03 23:02:53.473944 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.473873 6234 scope.go:122] "RemoveContainer" containerID="59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a" Nov 03 23:02:53.483658 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.483633 6234 scope.go:122] "RemoveContainer" containerID="ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7" Nov 03 23:02:53.491665 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.491640 6234 scope.go:122] "RemoveContainer" containerID="59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a" Nov 03 23:02:53.491913 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:02:53.491896 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a\": container with ID starting with 59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a not found: ID does not exist" containerID="59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a" Nov 03 23:02:53.491963 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.491921 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a"} err="failed to get container status \"59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a\": rpc error: code = NotFound desc = could not find container \"59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a\": container with ID starting with 59419c47444fd374c25f2051405fc6bd2d724a09b838e5d670df34051fde499a not found: ID does not exist" Nov 03 23:02:53.491963 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.491938 6234 scope.go:122] "RemoveContainer" containerID="ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7" Nov 03 23:02:53.492147 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:02:53.492131 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7\": container with ID starting with ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7 not found: ID does not exist" containerID="ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7" Nov 03 23:02:53.492192 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.492151 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7"} err="failed to get container status \"ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7\": rpc error: code = NotFound desc = could not find container \"ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7\": container with ID starting with ee4cc5f329dcc0a58661291421d59635d96a23e5820e832170d21f1afbffbfb7 not found: ID does not exist" Nov 03 23:02:53.627741 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.627663 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6"] Nov 03 23:02:53.631345 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:53.631321 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe3af2-openshift-default-75fb49b585d45g6"] Nov 03 23:02:54.703977 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:54.703943 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" path="/var/lib/kubelet/pods/d78ced59-6b45-4dc3-8adf-a7bc342bba0e/volumes" Nov 03 23:02:55.916570 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:02:55.916537 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" podUID="a4a542be-eda5-4ccb-8669-637023d7f739" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:02:58.996299 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:02:58.996263 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:03:01.503645 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.503589 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:03:01.513515 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.513469 6234 generic.go:356] "Generic (PLEG): container finished" podID="a4a542be-eda5-4ccb-8669-637023d7f739" containerID="8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1" exitCode=0 Nov 03 23:03:01.513655 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.513525 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" Nov 03 23:03:01.513655 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.513550 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" event={"ID":"a4a542be-eda5-4ccb-8669-637023d7f739","Type":"ContainerDied","Data":"8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1"} Nov 03 23:03:01.513655 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.513626 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl" event={"ID":"a4a542be-eda5-4ccb-8669-637023d7f739","Type":"ContainerDied","Data":"b5160c14d2fa6658497e1edab49b78f448a87fb4bdeec9e01647811a8aa56512"} Nov 03 23:03:01.513655 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.513652 6234 scope.go:122] "RemoveContainer" containerID="8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1" Nov 03 23:03:01.524082 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.524058 6234 scope.go:122] "RemoveContainer" containerID="8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1" Nov 03 23:03:01.524516 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:03:01.524476 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1\": container with ID starting with 8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1 not found: ID does not exist" containerID="8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1" Nov 03 23:03:01.524613 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.524530 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1"} err="failed to get container status \"8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1\": rpc error: code = NotFound desc = could not find container \"8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1\": container with ID starting with 8cccb69a2b537c640fd747fa67a473f0b83e1170a628de551a904b5a11f43fb1 not found: ID does not exist" Nov 03 23:03:01.554726 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.554703 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a542be-eda5-4ccb-8669-637023d7f739-kube-api-access-42dl6\" (UniqueName: \"kubernetes.io/projected/a4a542be-eda5-4ccb-8669-637023d7f739-kube-api-access-42dl6\") pod \"a4a542be-eda5-4ccb-8669-637023d7f739\" (UID: \"a4a542be-eda5-4ccb-8669-637023d7f739\") " Nov 03 23:03:01.554813 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.554784 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a542be-eda5-4ccb-8669-637023d7f739-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a542be-eda5-4ccb-8669-637023d7f739-plugins-config-volume\") pod \"a4a542be-eda5-4ccb-8669-637023d7f739\" (UID: \"a4a542be-eda5-4ccb-8669-637023d7f739\") " Nov 03 23:03:01.555109 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.555090 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a542be-eda5-4ccb-8669-637023d7f739-plugins-config-volume" pod "a4a542be-eda5-4ccb-8669-637023d7f739" (UID: "a4a542be-eda5-4ccb-8669-637023d7f739"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:01.556701 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.556681 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a542be-eda5-4ccb-8669-637023d7f739-kube-api-access-42dl6" pod "a4a542be-eda5-4ccb-8669-637023d7f739" (UID: "a4a542be-eda5-4ccb-8669-637023d7f739"). InnerVolumeSpecName "kube-api-access-42dl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:01.655853 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.655818 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42dl6\" (UniqueName: \"kubernetes.io/projected/a4a542be-eda5-4ccb-8669-637023d7f739-kube-api-access-42dl6\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:01.655853 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.655846 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a542be-eda5-4ccb-8669-637023d7f739-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:01.926281 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.926247 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl"] Nov 03 23:03:01.930533 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:01.930505 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe3af2-79dd7rngcl"] Nov 03 23:03:02.702751 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:03:02.702717 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a542be-eda5-4ccb-8669-637023d7f739" path="/var/lib/kubelet/pods/a4a542be-eda5-4ccb-8669-637023d7f739/volumes" Nov 03 23:03:28.996993 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:03:28.996950 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:03:58.995559 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:03:58.995488 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:04:28.995657 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:04:28.995615 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:04:31.982290 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:31.982255 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:04:34.327662 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:34.327614 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:04:34.328434 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:34.328393 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-bqdwt" podUID="fce20530-f3fd-4d1a-9dcd-151248db648a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:04:34.724942 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:34.724586 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:04:34.724942 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:34.724672 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-w4b5m" podUID="7f3294b5-bf10-41a8-b126-6393399d4b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:04:35.011683 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:35.011589 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:04:35.011965 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:35.011933 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-zh7jb" podUID="8e40f9f4-42b6-4c99-8b00-1ba515d7cf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:04:35.075621 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:35.075583 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-64665 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:04:35.075890 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:35.075862 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-64665" podUID="7468570c-3547-4aa4-9d7e-c4f4aad4c395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:04:49.296305 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:49.296271 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:04:58.989296 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:04:58.989252 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:04:58.997696 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:04:58.997661 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:05:18.482772 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.482681 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:05:18.550323 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550292 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl"] Nov 03 23:05:18.550744 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550729 6234 cpu_manager.go:414] "containerMap: removing container" podUID="8156999a-b200-4d93-b27b-11856682a544" containerName="main" Nov 03 23:05:18.550797 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550746 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8156999a-b200-4d93-b27b-11856682a544" containerName="main" Nov 03 23:05:18.550797 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550765 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" Nov 03 23:05:18.550797 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550771 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" Nov 03 23:05:18.550797 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550784 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a4a542be-eda5-4ccb-8669-637023d7f739" containerName="payload-pre-processing" Nov 03 23:05:18.550797 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550790 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a542be-eda5-4ccb-8669-637023d7f739" containerName="payload-pre-processing" Nov 03 23:05:18.550961 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550856 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" Nov 03 23:05:18.550961 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550865 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a542be-eda5-4ccb-8669-637023d7f739" containerName="payload-pre-processing" Nov 03 23:05:18.550961 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550874 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" Nov 03 23:05:18.550961 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.550881 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="8156999a-b200-4d93-b27b-11856682a544" containerName="main" Nov 03 23:05:18.553754 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.553733 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.604684 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.604642 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-28c170-openshift-default-dockercfg-wxrcc\"" Nov 03 23:05:18.652100 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.652066 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl"] Nov 03 23:05:18.680946 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.680912 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-token\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.680946 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.680951 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-data\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.681157 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.680973 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-credential-socket\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.681157 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.681091 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-socket\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.681157 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.681122 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-envoy\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.681157 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.681143 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-podinfo\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.681305 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.681233 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f52288-aba6-44d2-aadc-61f9610fe0da-istiod-ca-cert\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.681305 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.681260 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-certs\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.681305 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.681290 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mnf\" (UniqueName: \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-kube-api-access-j8mnf\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.782552 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782453 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-token\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.782552 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782490 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-data\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.782552 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782545 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-credential-socket\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.782813 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782577 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-socket\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.782813 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782594 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-envoy\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.782813 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782620 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-podinfo\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.782813 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782671 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f52288-aba6-44d2-aadc-61f9610fe0da-istiod-ca-cert\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.782813 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782704 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-certs\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.782813 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782750 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mnf\" (UniqueName: \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-kube-api-access-j8mnf\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.783140 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782902 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-data\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.783140 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.782987 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-credential-socket\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.783140 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.783049 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-socket\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.783265 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.783183 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-certs\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.783357 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.783338 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f52288-aba6-44d2-aadc-61f9610fe0da-istiod-ca-cert\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.784861 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.784836 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-envoy\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.785337 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.785275 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-podinfo\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.835273 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.835244 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-token\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.835483 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.835459 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mnf\" (UniqueName: \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-kube-api-access-j8mnf\") pod \"e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:18.865878 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:18.865854 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:19.291644 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:19.291611 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl"] Nov 03 23:05:19.297435 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:05:19.297404 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f52288_aba6_44d2_aadc_61f9610fe0da.slice/crio-b8ae99e398175c0632fec0a24e4b6d77ef859d18f1a77c83729eff312ada1a27 WatchSource:0}: Error finding container b8ae99e398175c0632fec0a24e4b6d77ef859d18f1a77c83729eff312ada1a27: Status 404 returned error can't find the container with id b8ae99e398175c0632fec0a24e4b6d77ef859d18f1a77c83729eff312ada1a27 Nov 03 23:05:19.299545 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:19.299490 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:05:19.299656 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:19.299621 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:05:19.299779 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:19.299666 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:05:20.199915 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:20.199877 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" event={"ID":"56f52288-aba6-44d2-aadc-61f9610fe0da","Type":"ContainerStarted","Data":"90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781"} Nov 03 23:05:20.200365 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:20.199929 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" event={"ID":"56f52288-aba6-44d2-aadc-61f9610fe0da","Type":"ContainerStarted","Data":"b8ae99e398175c0632fec0a24e4b6d77ef859d18f1a77c83729eff312ada1a27"} Nov 03 23:05:20.357231 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:20.357179 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" podStartSLOduration=2.357162292 podStartE2EDuration="2.357162292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:20.355851671 +0000 UTC m=+3439.768037688" watchObservedRunningTime="2026-11-03 23:05:20.357162292 +0000 UTC m=+3439.769348308" Nov 03 23:05:20.866900 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:20.866866 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:20.871817 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:20.871792 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:21.210386 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:21.208148 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:21.211551 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:21.211526 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:27.131411 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.131370 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5"] Nov 03 23:05:27.132032 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.132012 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" Nov 03 23:05:27.132157 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.132033 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78ced59-6b45-4dc3-8adf-a7bc342bba0e" containerName="istio-proxy" Nov 03 23:05:27.135964 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.135945 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:05:27.181058 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.181029 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5"] Nov 03 23:05:27.225969 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.225936 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-28c170\"" Nov 03 23:05:27.226129 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.226096 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5b493930-dockercfg-cb77b\"" Nov 03 23:05:27.270628 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.270601 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fqrz\" (UniqueName: \"kubernetes.io/projected/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-kube-api-access-6fqrz\") pod \"payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5\" (UID: \"27dc0e9a-9871-463d-a836-2f1ce58ab5b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:05:27.270795 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.270780 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5\" (UID: \"27dc0e9a-9871-463d-a836-2f1ce58ab5b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:05:27.371904 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.371864 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5\" (UID: \"27dc0e9a-9871-463d-a836-2f1ce58ab5b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:05:27.372064 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.371954 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fqrz\" (UniqueName: \"kubernetes.io/projected/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-kube-api-access-6fqrz\") pod \"payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5\" (UID: \"27dc0e9a-9871-463d-a836-2f1ce58ab5b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:05:27.372680 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.372657 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5\" (UID: \"27dc0e9a-9871-463d-a836-2f1ce58ab5b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:05:27.426095 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.426007 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fqrz\" (UniqueName: \"kubernetes.io/projected/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-kube-api-access-6fqrz\") pod \"payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5\" (UID: \"27dc0e9a-9871-463d-a836-2f1ce58ab5b4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:05:27.450548 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.450515 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:05:27.817204 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:27.817167 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5"] Nov 03 23:05:27.822964 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:05:27.822934 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27dc0e9a_9871_463d_a836_2f1ce58ab5b4.slice/crio-56c4cb25bb9b3a87a9c7ddc81436590da9ab6a5c1c43f1a8aab33bc6834ee62d WatchSource:0}: Error finding container 56c4cb25bb9b3a87a9c7ddc81436590da9ab6a5c1c43f1a8aab33bc6834ee62d: Status 404 returned error can't find the container with id 56c4cb25bb9b3a87a9c7ddc81436590da9ab6a5c1c43f1a8aab33bc6834ee62d Nov 03 23:05:28.239027 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:28.238991 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" event={"ID":"27dc0e9a-9871-463d-a836-2f1ce58ab5b4","Type":"ContainerStarted","Data":"31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22"} Nov 03 23:05:28.239454 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:28.239047 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" event={"ID":"27dc0e9a-9871-463d-a836-2f1ce58ab5b4","Type":"ContainerStarted","Data":"56c4cb25bb9b3a87a9c7ddc81436590da9ab6a5c1c43f1a8aab33bc6834ee62d"} Nov 03 23:05:28.239454 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:28.239076 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:05:28.346950 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:28.346896 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" podStartSLOduration=1.346882762 podStartE2EDuration="1.346882762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:28.345650672 +0000 UTC m=+3447.757836692" watchObservedRunningTime="2026-11-03 23:05:28.346882762 +0000 UTC m=+3447.759068826" Nov 03 23:05:28.996566 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:05:28.996530 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:05:38.361434 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:38.361398 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:05:39.246633 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:39.246607 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:05:48.332648 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:48.332555 6234 generic.go:356] "Generic (PLEG): container finished" podID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerID="90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781" exitCode=0 Nov 03 23:05:48.333049 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:48.332636 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" event={"ID":"56f52288-aba6-44d2-aadc-61f9610fe0da","Type":"ContainerDied","Data":"90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781"} Nov 03 23:05:48.333049 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:48.333014 6234 scope.go:122] "RemoveContainer" containerID="90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781" Nov 03 23:05:48.334276 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:48.334252 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:05:48.334339 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:48.334315 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:05:48.334375 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:48.334348 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:05:49.365779 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:49.365747 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" event={"ID":"56f52288-aba6-44d2-aadc-61f9610fe0da","Type":"ContainerStarted","Data":"dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e"} Nov 03 23:05:49.866130 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:49.866097 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:49.867412 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:49.867383 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 23:05:49.867581 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:49.867456 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 23:05:50.867042 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:50.866998 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 23:05:50.867461 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:50.867075 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 23:05:51.208487 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:51.208456 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:51.867035 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:51.867001 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 23:05:51.867460 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:51.867058 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 23:05:52.558419 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.558381 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb"] Nov 03 23:05:52.566177 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.566142 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.613484 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.613446 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb"] Nov 03 23:05:52.636107 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.636079 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-model-cache\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.636265 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.636111 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-dshm\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.636265 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.636151 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-home\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.636265 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.636221 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h7jm\" (UniqueName: \"kubernetes.io/projected/bf8adc4d-053e-4928-8962-a765933806de-kube-api-access-7h7jm\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.636380 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.636285 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8adc4d-053e-4928-8962-a765933806de-tls-certs\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.636380 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.636311 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-tmp-dir\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.663521 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.662731 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-28c170\"/\"openshift-service-ca.crt\"" Nov 03 23:05:52.663521 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.662871 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-28c170\"/\"default-dockercfg-6th22\"" Nov 03 23:05:52.663521 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.662988 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-28c170\"/\"simulator-single-main-28c170-kserve-self-signed-certs\"" Nov 03 23:05:52.663521 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.663050 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-28c170\"/\"kube-root-ca.crt\"" Nov 03 23:05:52.737368 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.737331 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-model-cache\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.737368 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.737365 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-dshm\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.737651 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.737481 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-home\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.737651 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.737519 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h7jm\" (UniqueName: \"kubernetes.io/projected/bf8adc4d-053e-4928-8962-a765933806de-kube-api-access-7h7jm\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.737651 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.737542 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8adc4d-053e-4928-8962-a765933806de-tls-certs\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.737651 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.737562 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-tmp-dir\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.737887 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.737857 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-model-cache\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.738152 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.738123 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-tmp-dir\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.738152 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.738137 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-home\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.748797 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.748748 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-dshm\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.754785 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.754733 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8adc4d-053e-4928-8962-a765933806de-tls-certs\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.791581 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.791549 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h7jm\" (UniqueName: \"kubernetes.io/projected/bf8adc4d-053e-4928-8962-a765933806de-kube-api-access-7h7jm\") pod \"simulator-single-main-28c170-kserve-6f4658998c-lszrb\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:52.868155 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.867100 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 23:05:52.868155 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.867183 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 23:05:52.878527 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:52.878482 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:53.259512 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:53.249082 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb"] Nov 03 23:05:53.267133 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:05:53.267105 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf8adc4d_053e_4928_8962_a765933806de.slice/crio-1db72f009df6a732f0b688a1bb450fdd7980e3daca2d199a5e6cc53a36836570 WatchSource:0}: Error finding container 1db72f009df6a732f0b688a1bb450fdd7980e3daca2d199a5e6cc53a36836570: Status 404 returned error can't find the container with id 1db72f009df6a732f0b688a1bb450fdd7980e3daca2d199a5e6cc53a36836570 Nov 03 23:05:53.386970 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:53.386927 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" event={"ID":"bf8adc4d-053e-4928-8962-a765933806de","Type":"ContainerStarted","Data":"b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42"} Nov 03 23:05:53.386970 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:53.386974 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" event={"ID":"bf8adc4d-053e-4928-8962-a765933806de","Type":"ContainerStarted","Data":"1db72f009df6a732f0b688a1bb450fdd7980e3daca2d199a5e6cc53a36836570"} Nov 03 23:05:53.866257 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:53.866223 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 23:05:53.866479 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:53.866285 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 23:05:53.879002 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:53.878963 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:05:54.870766 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:54.870738 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:54.871756 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:54.871734 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:05:55.028019 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:55.027967 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" podStartSLOduration=3.027953589 podStartE2EDuration="3.027953589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:53.493978597 +0000 UTC m=+3472.906164614" watchObservedRunningTime="2026-11-03 23:05:55.027953589 +0000 UTC m=+3474.440139605" Nov 03 23:05:55.580991 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:05:55.580954 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:05:58.996357 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:05:58.996322 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:06:02.879295 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:06:02.879261 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:06:02.892084 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:06:02.892057 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:06:03.446577 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:06:03.446545 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:06:22.187137 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:06:22.187094 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:06:28.996239 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:06:28.996195 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:06:50.088773 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:06:50.088730 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:06:53.183179 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:06:53.183137 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb"] Nov 03 23:06:58.996775 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:06:58.996732 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:06:59.761422 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:06:59.761390 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5"] Nov 03 23:07:04.275296 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:04.275261 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:07:04.762609 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:04.762558 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" podUID="27dc0e9a-9871-463d-a836-2f1ce58ab5b4" containerName="payload-processing" containerID="cri-o://31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22" gracePeriod=25 Nov 03 23:07:05.009453 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.009426 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:07:05.104147 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.104078 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-kube-api-access-6fqrz\" (UniqueName: \"kubernetes.io/projected/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-kube-api-access-6fqrz\") pod \"27dc0e9a-9871-463d-a836-2f1ce58ab5b4\" (UID: \"27dc0e9a-9871-463d-a836-2f1ce58ab5b4\") " Nov 03 23:07:05.104147 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.104124 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-plugins-config-volume\") pod \"27dc0e9a-9871-463d-a836-2f1ce58ab5b4\" (UID: \"27dc0e9a-9871-463d-a836-2f1ce58ab5b4\") " Nov 03 23:07:05.104570 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.104545 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-plugins-config-volume" pod "27dc0e9a-9871-463d-a836-2f1ce58ab5b4" (UID: "27dc0e9a-9871-463d-a836-2f1ce58ab5b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:05.106178 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.106153 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-kube-api-access-6fqrz" pod "27dc0e9a-9871-463d-a836-2f1ce58ab5b4" (UID: "27dc0e9a-9871-463d-a836-2f1ce58ab5b4"). InnerVolumeSpecName "kube-api-access-6fqrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:05.205093 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.205058 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fqrz\" (UniqueName: \"kubernetes.io/projected/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-kube-api-access-6fqrz\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:05.205093 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.205088 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27dc0e9a-9871-463d-a836-2f1ce58ab5b4-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:05.734244 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.734209 6234 generic.go:356] "Generic (PLEG): container finished" podID="27dc0e9a-9871-463d-a836-2f1ce58ab5b4" containerID="31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22" exitCode=0 Nov 03 23:07:05.734697 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.734245 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" Nov 03 23:07:05.734697 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.734289 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" event={"ID":"27dc0e9a-9871-463d-a836-2f1ce58ab5b4","Type":"ContainerDied","Data":"31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22"} Nov 03 23:07:05.734697 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.734336 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5" event={"ID":"27dc0e9a-9871-463d-a836-2f1ce58ab5b4","Type":"ContainerDied","Data":"56c4cb25bb9b3a87a9c7ddc81436590da9ab6a5c1c43f1a8aab33bc6834ee62d"} Nov 03 23:07:05.734697 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.734357 6234 scope.go:122] "RemoveContainer" containerID="31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22" Nov 03 23:07:05.743375 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.743357 6234 scope.go:122] "RemoveContainer" containerID="31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22" Nov 03 23:07:05.743655 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:07:05.743634 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22\": container with ID starting with 31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22 not found: ID does not exist" containerID="31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22" Nov 03 23:07:05.743729 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.743670 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22"} err="failed to get container status \"31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22\": rpc error: code = NotFound desc = could not find container \"31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22\": container with ID starting with 31a85fe27a408b549bf743650e605107e9ce382b9ac5e2fa1c4b43488521fd22 not found: ID does not exist" Nov 03 23:07:05.887037 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.887005 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5"] Nov 03 23:07:05.933380 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:05.933341 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-28c170-79d9c796597vc5"] Nov 03 23:07:06.703653 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:06.703620 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27dc0e9a-9871-463d-a836-2f1ce58ab5b4" path="/var/lib/kubelet/pods/27dc0e9a-9871-463d-a836-2f1ce58ab5b4/volumes" Nov 03 23:07:08.203385 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.203345 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" podUID="bf8adc4d-053e-4928-8962-a765933806de" containerName="main" containerID="cri-o://b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42" gracePeriod=45 Nov 03 23:07:08.452067 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.452035 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:07:08.539148 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.539054 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8adc4d-053e-4928-8962-a765933806de-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8adc4d-053e-4928-8962-a765933806de-tls-certs\") pod \"bf8adc4d-053e-4928-8962-a765933806de\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " Nov 03 23:07:08.539148 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.539119 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-dshm\") pod \"bf8adc4d-053e-4928-8962-a765933806de\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " Nov 03 23:07:08.539148 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.539138 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-home\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-home\") pod \"bf8adc4d-053e-4928-8962-a765933806de\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " Nov 03 23:07:08.539448 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.539204 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8adc4d-053e-4928-8962-a765933806de-kube-api-access-7h7jm\" (UniqueName: \"kubernetes.io/projected/bf8adc4d-053e-4928-8962-a765933806de-kube-api-access-7h7jm\") pod \"bf8adc4d-053e-4928-8962-a765933806de\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " Nov 03 23:07:08.539448 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.539223 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-model-cache\") pod \"bf8adc4d-053e-4928-8962-a765933806de\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " Nov 03 23:07:08.539448 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.539305 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-tmp-dir\") pod \"bf8adc4d-053e-4928-8962-a765933806de\" (UID: \"bf8adc4d-053e-4928-8962-a765933806de\") " Nov 03 23:07:08.539659 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.539477 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-home" pod "bf8adc4d-053e-4928-8962-a765933806de" (UID: "bf8adc4d-053e-4928-8962-a765933806de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:08.539715 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.539666 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-model-cache" pod "bf8adc4d-053e-4928-8962-a765933806de" (UID: "bf8adc4d-053e-4928-8962-a765933806de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:08.539715 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.539681 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-tmp-dir" pod "bf8adc4d-053e-4928-8962-a765933806de" (UID: "bf8adc4d-053e-4928-8962-a765933806de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:08.539814 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.539729 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:08.541602 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.541577 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8adc4d-053e-4928-8962-a765933806de-tls-certs" pod "bf8adc4d-053e-4928-8962-a765933806de" (UID: "bf8adc4d-053e-4928-8962-a765933806de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:08.541602 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.541581 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-dshm" pod "bf8adc4d-053e-4928-8962-a765933806de" (UID: "bf8adc4d-053e-4928-8962-a765933806de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:08.541754 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.541658 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8adc4d-053e-4928-8962-a765933806de-kube-api-access-7h7jm" pod "bf8adc4d-053e-4928-8962-a765933806de" (UID: "bf8adc4d-053e-4928-8962-a765933806de"). InnerVolumeSpecName "kube-api-access-7h7jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:08.640828 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.640798 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:08.640828 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.640824 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8adc4d-053e-4928-8962-a765933806de-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:08.640828 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.640832 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:08.641056 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.640841 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h7jm\" (UniqueName: \"kubernetes.io/projected/bf8adc4d-053e-4928-8962-a765933806de-kube-api-access-7h7jm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:08.641056 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.640851 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8adc4d-053e-4928-8962-a765933806de-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:08.750910 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.750877 6234 generic.go:356] "Generic (PLEG): container finished" podID="bf8adc4d-053e-4928-8962-a765933806de" containerID="b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42" exitCode=0 Nov 03 23:07:08.751092 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.750917 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" Nov 03 23:07:08.751092 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.750960 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" event={"ID":"bf8adc4d-053e-4928-8962-a765933806de","Type":"ContainerDied","Data":"b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42"} Nov 03 23:07:08.751092 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.751005 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" event={"ID":"bf8adc4d-053e-4928-8962-a765933806de","Type":"ContainerDied","Data":"1db72f009df6a732f0b688a1bb450fdd7980e3daca2d199a5e6cc53a36836570"} Nov 03 23:07:08.751092 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.751026 6234 scope.go:122] "RemoveContainer" containerID="b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42" Nov 03 23:07:08.761903 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.761874 6234 scope.go:122] "RemoveContainer" containerID="b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42" Nov 03 23:07:08.762351 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:07:08.762327 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42\": container with ID starting with b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42 not found: ID does not exist" containerID="b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42" Nov 03 23:07:08.762411 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.762361 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42"} err="failed to get container status \"b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42\": rpc error: code = NotFound desc = could not find container \"b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42\": container with ID starting with b9d549cc7c00a5edd49e555649a47f24130ca54fbb1e04dcddfe7a15e1327c42 not found: ID does not exist" Nov 03 23:07:08.903304 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.903206 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb"] Nov 03 23:07:08.905856 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:08.905832 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb"] Nov 03 23:07:09.436630 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:09.436570 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-28c170/simulator-single-main-28c170-kserve-6f4658998c-lszrb" podUID="bf8adc4d-053e-4928-8962-a765933806de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded" Nov 03 23:07:10.703912 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:10.703878 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8adc4d-053e-4928-8962-a765933806de" path="/var/lib/kubelet/pods/bf8adc4d-053e-4928-8962-a765933806de/volumes" Nov 03 23:07:11.467846 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:11.467810 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl"] Nov 03 23:07:11.468109 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:11.468084 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" containerID="cri-o://dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e" gracePeriod=30 Nov 03 23:07:16.728152 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.728128 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:07:16.733356 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733296 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-credential-socket\") pod \"56f52288-aba6-44d2-aadc-61f9610fe0da\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " Nov 03 23:07:16.733356 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733337 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-envoy\") pod \"56f52288-aba6-44d2-aadc-61f9610fe0da\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " Nov 03 23:07:16.733565 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733366 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-data\") pod \"56f52288-aba6-44d2-aadc-61f9610fe0da\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " Nov 03 23:07:16.733565 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733446 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-podinfo\") pod \"56f52288-aba6-44d2-aadc-61f9610fe0da\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " Nov 03 23:07:16.733565 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733479 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f52288-aba6-44d2-aadc-61f9610fe0da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f52288-aba6-44d2-aadc-61f9610fe0da-istiod-ca-cert\") pod \"56f52288-aba6-44d2-aadc-61f9610fe0da\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " Nov 03 23:07:16.733565 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733526 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-socket\") pod \"56f52288-aba6-44d2-aadc-61f9610fe0da\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " Nov 03 23:07:16.733565 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733538 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-credential-socket" pod "56f52288-aba6-44d2-aadc-61f9610fe0da" (UID: "56f52288-aba6-44d2-aadc-61f9610fe0da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:16.733832 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733567 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-token\" (UniqueName: \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-token\") pod \"56f52288-aba6-44d2-aadc-61f9610fe0da\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " Nov 03 23:07:16.733832 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733662 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-kube-api-access-j8mnf\" (UniqueName: \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-kube-api-access-j8mnf\") pod \"56f52288-aba6-44d2-aadc-61f9610fe0da\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " Nov 03 23:07:16.733832 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733716 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-certs\") pod \"56f52288-aba6-44d2-aadc-61f9610fe0da\" (UID: \"56f52288-aba6-44d2-aadc-61f9610fe0da\") " Nov 03 23:07:16.733832 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.733743 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-data" pod "56f52288-aba6-44d2-aadc-61f9610fe0da" (UID: "56f52288-aba6-44d2-aadc-61f9610fe0da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:16.734064 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.734041 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-certs" pod "56f52288-aba6-44d2-aadc-61f9610fe0da" (UID: "56f52288-aba6-44d2-aadc-61f9610fe0da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:16.734222 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.734174 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:16.734222 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.734188 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-socket" pod "56f52288-aba6-44d2-aadc-61f9610fe0da" (UID: "56f52288-aba6-44d2-aadc-61f9610fe0da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:16.734222 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.734201 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:16.734354 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.734291 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f52288-aba6-44d2-aadc-61f9610fe0da-istiod-ca-cert" pod "56f52288-aba6-44d2-aadc-61f9610fe0da" (UID: "56f52288-aba6-44d2-aadc-61f9610fe0da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:16.735672 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.735650 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-envoy" pod "56f52288-aba6-44d2-aadc-61f9610fe0da" (UID: "56f52288-aba6-44d2-aadc-61f9610fe0da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:16.735899 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.735884 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-kube-api-access-j8mnf" pod "56f52288-aba6-44d2-aadc-61f9610fe0da" (UID: "56f52288-aba6-44d2-aadc-61f9610fe0da"). InnerVolumeSpecName "kube-api-access-j8mnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:16.736083 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.736056 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-podinfo" pod "56f52288-aba6-44d2-aadc-61f9610fe0da" (UID: "56f52288-aba6-44d2-aadc-61f9610fe0da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:16.736179 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.736062 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-token" pod "56f52288-aba6-44d2-aadc-61f9610fe0da" (UID: "56f52288-aba6-44d2-aadc-61f9610fe0da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:16.790739 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.790705 6234 generic.go:356] "Generic (PLEG): container finished" podID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerID="dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e" exitCode=0 Nov 03 23:07:16.790885 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.790760 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" Nov 03 23:07:16.790885 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.790779 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" event={"ID":"56f52288-aba6-44d2-aadc-61f9610fe0da","Type":"ContainerDied","Data":"dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e"} Nov 03 23:07:16.790885 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.790832 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl" event={"ID":"56f52288-aba6-44d2-aadc-61f9610fe0da","Type":"ContainerDied","Data":"b8ae99e398175c0632fec0a24e4b6d77ef859d18f1a77c83729eff312ada1a27"} Nov 03 23:07:16.790885 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.790863 6234 scope.go:122] "RemoveContainer" containerID="dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e" Nov 03 23:07:16.799895 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.799873 6234 scope.go:122] "RemoveContainer" containerID="90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781" Nov 03 23:07:16.807583 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.807561 6234 scope.go:122] "RemoveContainer" containerID="dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e" Nov 03 23:07:16.807834 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:07:16.807812 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e\": container with ID starting with dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e not found: ID does not exist" containerID="dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e" Nov 03 23:07:16.807918 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.807846 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e"} err="failed to get container status \"dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e\": rpc error: code = NotFound desc = could not find container \"dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e\": container with ID starting with dd1d02506d44d8d666954263c039d117e23bea64ea35b3c330f09a3b090eaf9e not found: ID does not exist" Nov 03 23:07:16.807918 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.807870 6234 scope.go:122] "RemoveContainer" containerID="90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781" Nov 03 23:07:16.808105 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:07:16.808088 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781\": container with ID starting with 90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781 not found: ID does not exist" containerID="90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781" Nov 03 23:07:16.808144 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.808111 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781"} err="failed to get container status \"90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781\": rpc error: code = NotFound desc = could not find container \"90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781\": container with ID starting with 90c08a39ebc11ad6170fb554d3f9a1c6f306f09213f458aef205cec32ce0d781 not found: ID does not exist" Nov 03 23:07:16.835396 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.835375 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:16.835396 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.835396 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f52288-aba6-44d2-aadc-61f9610fe0da-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:16.835549 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.835406 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:16.835549 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.835416 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:16.835549 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.835425 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8mnf\" (UniqueName: \"kubernetes.io/projected/56f52288-aba6-44d2-aadc-61f9610fe0da-kube-api-access-j8mnf\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:16.835549 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.835435 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:16.835549 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.835443 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f52288-aba6-44d2-aadc-61f9610fe0da-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:16.992756 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:16.992673 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl"] Nov 03 23:07:17.006701 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:17.006667 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-28c170-openshift-default-6f7674b7fc8ghkl"] Nov 03 23:07:18.088794 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.088761 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl"] Nov 03 23:07:18.089172 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.089134 6234 cpu_manager.go:414] "containerMap: removing container" podUID="bf8adc4d-053e-4928-8962-a765933806de" containerName="main" Nov 03 23:07:18.089172 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.089145 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8adc4d-053e-4928-8962-a765933806de" containerName="main" Nov 03 23:07:18.089172 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.089162 6234 cpu_manager.go:414] "containerMap: removing container" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" Nov 03 23:07:18.089172 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.089167 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" Nov 03 23:07:18.089298 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.089180 6234 cpu_manager.go:414] "containerMap: removing container" podUID="27dc0e9a-9871-463d-a836-2f1ce58ab5b4" containerName="payload-processing" Nov 03 23:07:18.089298 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.089185 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27dc0e9a-9871-463d-a836-2f1ce58ab5b4" containerName="payload-processing" Nov 03 23:07:18.089298 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.089256 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" Nov 03 23:07:18.089298 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.089264 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="27dc0e9a-9871-463d-a836-2f1ce58ab5b4" containerName="payload-processing" Nov 03 23:07:18.089298 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.089274 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8adc4d-053e-4928-8962-a765933806de" containerName="main" Nov 03 23:07:18.093862 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.093842 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.145172 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.145136 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-model-cache\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.145351 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.145205 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-home\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.145351 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.145317 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hzf\" (UniqueName: \"kubernetes.io/projected/b4950089-981a-4b05-b96f-7a9e9ee65924-kube-api-access-n4hzf\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.145351 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.145343 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-tmp-dir\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.145562 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.145377 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4950089-981a-4b05-b96f-7a9e9ee65924-tls-certs\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.145562 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.145543 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-dshm\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.146391 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.146369 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl"] Nov 03 23:07:18.191000 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.190971 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5240e263-kserve-self-signed-certs\"" Nov 03 23:07:18.246387 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.246350 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hzf\" (UniqueName: \"kubernetes.io/projected/b4950089-981a-4b05-b96f-7a9e9ee65924-kube-api-access-n4hzf\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.246604 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.246397 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-tmp-dir\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.246604 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.246457 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4950089-981a-4b05-b96f-7a9e9ee65924-tls-certs\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.246604 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.246526 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-dshm\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.246604 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.246554 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-model-cache\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.246838 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.246629 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-home\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.247066 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.247039 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-model-cache\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.247121 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.247067 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-home\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.247635 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.247613 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-tmp-dir\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.249809 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.249785 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-dshm\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.250866 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.250838 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4950089-981a-4b05-b96f-7a9e9ee65924-tls-certs\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.303151 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.303122 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hzf\" (UniqueName: \"kubernetes.io/projected/b4950089-981a-4b05-b96f-7a9e9ee65924-kube-api-access-n4hzf\") pod \"e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.405330 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.405255 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:18.703569 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.703527 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" path="/var/lib/kubelet/pods/56f52288-aba6-44d2-aadc-61f9610fe0da/volumes" Nov 03 23:07:18.771921 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.771889 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl"] Nov 03 23:07:18.778361 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:07:18.778330 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4950089_981a_4b05_b96f_7a9e9ee65924.slice/crio-9bcc9dcc750ae6c451d611e0b83b3186abd608d3e9da1d175f5e604dc19e68a4 WatchSource:0}: Error finding container 9bcc9dcc750ae6c451d611e0b83b3186abd608d3e9da1d175f5e604dc19e68a4: Status 404 returned error can't find the container with id 9bcc9dcc750ae6c451d611e0b83b3186abd608d3e9da1d175f5e604dc19e68a4 Nov 03 23:07:18.802319 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:18.802292 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" event={"ID":"b4950089-981a-4b05-b96f-7a9e9ee65924","Type":"ContainerStarted","Data":"9bcc9dcc750ae6c451d611e0b83b3186abd608d3e9da1d175f5e604dc19e68a4"} Nov 03 23:07:19.809197 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:19.809159 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" event={"ID":"b4950089-981a-4b05-b96f-7a9e9ee65924","Type":"ContainerStarted","Data":"b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264"} Nov 03 23:07:19.918287 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:19.918239 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" podStartSLOduration=1.9182268630000001 podStartE2EDuration="1.918226863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:19.917880883 +0000 UTC m=+3559.330066910" watchObservedRunningTime="2026-11-03 23:07:19.918226863 +0000 UTC m=+3559.330412877" Nov 03 23:07:20.406076 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:20.406044 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:22.905095 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:22.905053 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:07:28.406343 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:28.406299 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:28.418920 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:28.418897 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:28.876011 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:07:28.875982 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:07:29.000897 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:07:29.000854 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:07:58.995961 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:07:58.995924 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:08:01.186720 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.186690 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p"] Nov 03 23:08:01.187193 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.187100 6234 cpu_manager.go:414] "containerMap: removing container" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" Nov 03 23:08:01.187193 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.187113 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" Nov 03 23:08:01.187276 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.187207 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f52288-aba6-44d2-aadc-61f9610fe0da" containerName="istio-proxy" Nov 03 23:08:01.190610 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.190583 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.244475 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.244442 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p"] Nov 03 23:08:01.262620 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.262594 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5240e263-kserve-self-signed-certs\"" Nov 03 23:08:01.295787 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.295761 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-model-cache\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.295971 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.295855 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-dshm\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.295971 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.295926 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzs7w\" (UniqueName: \"kubernetes.io/projected/d430a614-bb46-40d6-9d30-dc1bc658d164-kube-api-access-tzs7w\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.296079 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.296013 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-home\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.296079 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.296040 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-tmp-dir\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.296188 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.296095 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430a614-bb46-40d6-9d30-dc1bc658d164-tls-certs\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.396975 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.396941 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-dshm\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.397152 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.396982 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzs7w\" (UniqueName: \"kubernetes.io/projected/d430a614-bb46-40d6-9d30-dc1bc658d164-kube-api-access-tzs7w\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.397152 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.397100 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-home\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.397152 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.397131 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-tmp-dir\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.397309 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.397156 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430a614-bb46-40d6-9d30-dc1bc658d164-tls-certs\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.397309 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.397287 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-model-cache\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.397544 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.397517 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-home\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.397544 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.397533 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-tmp-dir\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.397703 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.397583 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-model-cache\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.399620 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.399599 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-dshm\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.399735 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.399714 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430a614-bb46-40d6-9d30-dc1bc658d164-tls-certs\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.450408 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.450344 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzs7w\" (UniqueName: \"kubernetes.io/projected/d430a614-bb46-40d6-9d30-dc1bc658d164-kube-api-access-tzs7w\") pod \"e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.504187 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.504158 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:01.870981 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:01.870893 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p"] Nov 03 23:08:01.877013 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:08:01.876983 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd430a614_bb46_40d6_9d30_dc1bc658d164.slice/crio-ce6c059e0bcf04d95fbb2aeb19547c753d2986247010f9da235b77cc68086d85 WatchSource:0}: Error finding container ce6c059e0bcf04d95fbb2aeb19547c753d2986247010f9da235b77cc68086d85: Status 404 returned error can't find the container with id ce6c059e0bcf04d95fbb2aeb19547c753d2986247010f9da235b77cc68086d85 Nov 03 23:08:02.025634 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:02.025593 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" event={"ID":"d430a614-bb46-40d6-9d30-dc1bc658d164","Type":"ContainerStarted","Data":"ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5"} Nov 03 23:08:02.025634 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:02.025642 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" event={"ID":"d430a614-bb46-40d6-9d30-dc1bc658d164","Type":"ContainerStarted","Data":"ce6c059e0bcf04d95fbb2aeb19547c753d2986247010f9da235b77cc68086d85"} Nov 03 23:08:02.132911 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:02.132862 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" podStartSLOduration=1.132848522 podStartE2EDuration="1.132848522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:08:02.132476651 +0000 UTC m=+3601.544662671" watchObservedRunningTime="2026-11-03 23:08:02.132848522 +0000 UTC m=+3601.545034544" Nov 03 23:08:02.504893 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:02.504863 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:06.074695 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:06.074664 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:08:11.504849 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:11.504770 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:11.517631 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:11.517605 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:12.129728 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:12.129695 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:08:28.996097 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:08:28.996047 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:08:36.803966 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:36.803930 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:08:48.343993 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:48.343957 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl"] Nov 03 23:08:52.599984 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:52.599945 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:08:55.204520 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:08:55.199156 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p"] Nov 03 23:08:58.996065 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:08:58.996022 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:09:03.363708 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.363668 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" podUID="b4950089-981a-4b05-b96f-7a9e9ee65924" containerName="main" containerID="cri-o://b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264" gracePeriod=45 Nov 03 23:09:03.608987 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.608963 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:09:03.747935 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.747900 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-tmp-dir\") pod \"b4950089-981a-4b05-b96f-7a9e9ee65924\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " Nov 03 23:09:03.748128 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.748001 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4950089-981a-4b05-b96f-7a9e9ee65924-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4950089-981a-4b05-b96f-7a9e9ee65924-tls-certs\") pod \"b4950089-981a-4b05-b96f-7a9e9ee65924\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " Nov 03 23:09:03.748128 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.748037 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4950089-981a-4b05-b96f-7a9e9ee65924-kube-api-access-n4hzf\" (UniqueName: \"kubernetes.io/projected/b4950089-981a-4b05-b96f-7a9e9ee65924-kube-api-access-n4hzf\") pod \"b4950089-981a-4b05-b96f-7a9e9ee65924\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " Nov 03 23:09:03.748128 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.748086 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-dshm\") pod \"b4950089-981a-4b05-b96f-7a9e9ee65924\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " Nov 03 23:09:03.748306 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.748189 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-model-cache\") pod \"b4950089-981a-4b05-b96f-7a9e9ee65924\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " Nov 03 23:09:03.748306 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.748188 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-tmp-dir" pod "b4950089-981a-4b05-b96f-7a9e9ee65924" (UID: "b4950089-981a-4b05-b96f-7a9e9ee65924"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:03.748306 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.748222 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-home\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-home\") pod \"b4950089-981a-4b05-b96f-7a9e9ee65924\" (UID: \"b4950089-981a-4b05-b96f-7a9e9ee65924\") " Nov 03 23:09:03.748528 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.748459 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-model-cache" pod "b4950089-981a-4b05-b96f-7a9e9ee65924" (UID: "b4950089-981a-4b05-b96f-7a9e9ee65924"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:03.748601 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.748534 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-home" pod "b4950089-981a-4b05-b96f-7a9e9ee65924" (UID: "b4950089-981a-4b05-b96f-7a9e9ee65924"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:03.748646 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.748628 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:03.748693 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.748648 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:03.750302 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.750279 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-dshm" pod "b4950089-981a-4b05-b96f-7a9e9ee65924" (UID: "b4950089-981a-4b05-b96f-7a9e9ee65924"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:03.750398 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.750319 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4950089-981a-4b05-b96f-7a9e9ee65924-tls-certs" pod "b4950089-981a-4b05-b96f-7a9e9ee65924" (UID: "b4950089-981a-4b05-b96f-7a9e9ee65924"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:03.750637 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.750617 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4950089-981a-4b05-b96f-7a9e9ee65924-kube-api-access-n4hzf" pod "b4950089-981a-4b05-b96f-7a9e9ee65924" (UID: "b4950089-981a-4b05-b96f-7a9e9ee65924"). InnerVolumeSpecName "kube-api-access-n4hzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:03.849371 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.849347 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:03.849371 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.849372 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4950089-981a-4b05-b96f-7a9e9ee65924-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:03.849554 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.849381 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4hzf\" (UniqueName: \"kubernetes.io/projected/b4950089-981a-4b05-b96f-7a9e9ee65924-kube-api-access-n4hzf\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:03.849554 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:03.849390 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4950089-981a-4b05-b96f-7a9e9ee65924-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:04.457216 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:04.457174 6234 generic.go:356] "Generic (PLEG): container finished" podID="b4950089-981a-4b05-b96f-7a9e9ee65924" containerID="b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264" exitCode=0 Nov 03 23:09:04.457216 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:04.457217 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" Nov 03 23:09:04.457697 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:04.457260 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" event={"ID":"b4950089-981a-4b05-b96f-7a9e9ee65924","Type":"ContainerDied","Data":"b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264"} Nov 03 23:09:04.457697 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:04.457307 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl" event={"ID":"b4950089-981a-4b05-b96f-7a9e9ee65924","Type":"ContainerDied","Data":"9bcc9dcc750ae6c451d611e0b83b3186abd608d3e9da1d175f5e604dc19e68a4"} Nov 03 23:09:04.457697 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:04.457325 6234 scope.go:122] "RemoveContainer" containerID="b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264" Nov 03 23:09:04.466358 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:04.466340 6234 scope.go:122] "RemoveContainer" containerID="b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264" Nov 03 23:09:04.466644 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:09:04.466624 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264\": container with ID starting with b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264 not found: ID does not exist" containerID="b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264" Nov 03 23:09:04.466729 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:04.466652 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264"} err="failed to get container status \"b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264\": rpc error: code = NotFound desc = could not find container \"b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264\": container with ID starting with b0c399db6e148b3635e0eda97fca6926cd1ad35d1af657dddf126d87dd0b1264 not found: ID does not exist" Nov 03 23:09:04.609772 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:04.609744 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl"] Nov 03 23:09:04.611759 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:04.611740 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5240e263-kserve-b89bd9865-x2ntl"] Nov 03 23:09:04.703878 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:04.703847 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4950089-981a-4b05-b96f-7a9e9ee65924" path="/var/lib/kubelet/pods/b4950089-981a-4b05-b96f-7a9e9ee65924/volumes" Nov 03 23:09:06.295479 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:06.295439 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:09:10.633099 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:10.633056 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" podUID="d430a614-bb46-40d6-9d30-dc1bc658d164" containerName="main" containerID="cri-o://ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5" gracePeriod=45 Nov 03 23:09:10.883757 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:10.883698 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:09:11.021660 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.021615 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d430a614-bb46-40d6-9d30-dc1bc658d164-kube-api-access-tzs7w\" (UniqueName: \"kubernetes.io/projected/d430a614-bb46-40d6-9d30-dc1bc658d164-kube-api-access-tzs7w\") pod \"d430a614-bb46-40d6-9d30-dc1bc658d164\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " Nov 03 23:09:11.021856 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.021712 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-tmp-dir\") pod \"d430a614-bb46-40d6-9d30-dc1bc658d164\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " Nov 03 23:09:11.021856 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.021807 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d430a614-bb46-40d6-9d30-dc1bc658d164-tls-certs\" (UniqueName: \"kubernetes.io/secret/d430a614-bb46-40d6-9d30-dc1bc658d164-tls-certs\") pod \"d430a614-bb46-40d6-9d30-dc1bc658d164\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " Nov 03 23:09:11.021856 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.021850 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-home\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-home\") pod \"d430a614-bb46-40d6-9d30-dc1bc658d164\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " Nov 03 23:09:11.022027 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.021882 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-model-cache\") pod \"d430a614-bb46-40d6-9d30-dc1bc658d164\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " Nov 03 23:09:11.022027 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.021979 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-dshm\") pod \"d430a614-bb46-40d6-9d30-dc1bc658d164\" (UID: \"d430a614-bb46-40d6-9d30-dc1bc658d164\") " Nov 03 23:09:11.022027 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.021980 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-tmp-dir" pod "d430a614-bb46-40d6-9d30-dc1bc658d164" (UID: "d430a614-bb46-40d6-9d30-dc1bc658d164"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:11.022183 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.022124 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-home" pod "d430a614-bb46-40d6-9d30-dc1bc658d164" (UID: "d430a614-bb46-40d6-9d30-dc1bc658d164"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:11.022226 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.022193 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-model-cache" pod "d430a614-bb46-40d6-9d30-dc1bc658d164" (UID: "d430a614-bb46-40d6-9d30-dc1bc658d164"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:11.022422 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.022388 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-tmp-dir\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:11.022422 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.022421 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-home\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:11.022654 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.022430 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-model-cache\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:11.024046 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.024022 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-dshm" pod "d430a614-bb46-40d6-9d30-dc1bc658d164" (UID: "d430a614-bb46-40d6-9d30-dc1bc658d164"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:11.024357 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.024329 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d430a614-bb46-40d6-9d30-dc1bc658d164-tls-certs" pod "d430a614-bb46-40d6-9d30-dc1bc658d164" (UID: "d430a614-bb46-40d6-9d30-dc1bc658d164"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:11.024357 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.024345 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d430a614-bb46-40d6-9d30-dc1bc658d164-kube-api-access-tzs7w" pod "d430a614-bb46-40d6-9d30-dc1bc658d164" (UID: "d430a614-bb46-40d6-9d30-dc1bc658d164"). InnerVolumeSpecName "kube-api-access-tzs7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:11.123230 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.123191 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430a614-bb46-40d6-9d30-dc1bc658d164-tls-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:11.123230 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.123229 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430a614-bb46-40d6-9d30-dc1bc658d164-dshm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:11.123230 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.123238 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzs7w\" (UniqueName: \"kubernetes.io/projected/d430a614-bb46-40d6-9d30-dc1bc658d164-kube-api-access-tzs7w\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:11.491066 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.491018 6234 generic.go:356] "Generic (PLEG): container finished" podID="d430a614-bb46-40d6-9d30-dc1bc658d164" containerID="ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5" exitCode=0 Nov 03 23:09:11.491247 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.491138 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" event={"ID":"d430a614-bb46-40d6-9d30-dc1bc658d164","Type":"ContainerDied","Data":"ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5"} Nov 03 23:09:11.491247 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.491147 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" Nov 03 23:09:11.491247 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.491166 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p" event={"ID":"d430a614-bb46-40d6-9d30-dc1bc658d164","Type":"ContainerDied","Data":"ce6c059e0bcf04d95fbb2aeb19547c753d2986247010f9da235b77cc68086d85"} Nov 03 23:09:11.491247 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.491182 6234 scope.go:122] "RemoveContainer" containerID="ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5" Nov 03 23:09:11.500824 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.500801 6234 scope.go:122] "RemoveContainer" containerID="ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5" Nov 03 23:09:11.501133 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:09:11.501110 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5\": container with ID starting with ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5 not found: ID does not exist" containerID="ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5" Nov 03 23:09:11.501217 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.501148 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5"} err="failed to get container status \"ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5\": rpc error: code = NotFound desc = could not find container \"ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5\": container with ID starting with ca9ec3095a4b38976fe0905b5cfad5f790f9a29c3fbd9ea22959b714d0213ec5 not found: ID does not exist" Nov 03 23:09:11.643183 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.643152 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p"] Nov 03 23:09:11.646339 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:11.646314 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5240e263-kserve-9b7df8c7-4ls7p"] Nov 03 23:09:12.705686 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:12.703987 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d430a614-bb46-40d6-9d30-dc1bc658d164" path="/var/lib/kubelet/pods/d430a614-bb46-40d6-9d30-dc1bc658d164/volumes" Nov 03 23:09:18.188131 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:18.188089 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:09:28.996511 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:09:28.996456 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:09:41.694829 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:41.694793 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:09:42.422595 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.422562 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79bf796845-jkss4"] Nov 03 23:09:42.422990 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.422976 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b4950089-981a-4b05-b96f-7a9e9ee65924" containerName="main" Nov 03 23:09:42.423037 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.422993 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4950089-981a-4b05-b96f-7a9e9ee65924" containerName="main" Nov 03 23:09:42.423037 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.423006 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d430a614-bb46-40d6-9d30-dc1bc658d164" containerName="main" Nov 03 23:09:42.423037 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.423011 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d430a614-bb46-40d6-9d30-dc1bc658d164" containerName="main" Nov 03 23:09:42.423136 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.423093 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4950089-981a-4b05-b96f-7a9e9ee65924" containerName="main" Nov 03 23:09:42.423136 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.423101 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d430a614-bb46-40d6-9d30-dc1bc658d164" containerName="main" Nov 03 23:09:42.426203 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.426181 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.526049 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.526014 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79bf796845-jkss4"] Nov 03 23:09:42.542534 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.542490 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73570236-6aa6-4aa6-8067-c963ac57113e-maas-api-tls\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.542677 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.542594 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73570236-6aa6-4aa6-8067-c963ac57113e-openshift-service-ca\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.542677 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.542645 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73570236-6aa6-4aa6-8067-c963ac57113e-metrics-certs\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.542752 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.542688 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqdsr\" (UniqueName: \"kubernetes.io/projected/73570236-6aa6-4aa6-8067-c963ac57113e-kube-api-access-xqdsr\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.644095 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.644045 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqdsr\" (UniqueName: \"kubernetes.io/projected/73570236-6aa6-4aa6-8067-c963ac57113e-kube-api-access-xqdsr\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.644275 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.644143 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73570236-6aa6-4aa6-8067-c963ac57113e-maas-api-tls\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.644275 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.644177 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73570236-6aa6-4aa6-8067-c963ac57113e-openshift-service-ca\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.644275 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.644198 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73570236-6aa6-4aa6-8067-c963ac57113e-metrics-certs\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.644946 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.644920 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73570236-6aa6-4aa6-8067-c963ac57113e-openshift-service-ca\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.646694 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.646666 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73570236-6aa6-4aa6-8067-c963ac57113e-maas-api-tls\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.647168 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.646826 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73570236-6aa6-4aa6-8067-c963ac57113e-metrics-certs\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.696862 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.696833 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqdsr\" (UniqueName: \"kubernetes.io/projected/73570236-6aa6-4aa6-8067-c963ac57113e-kube-api-access-xqdsr\") pod \"maas-api-79bf796845-jkss4\" (UID: \"73570236-6aa6-4aa6-8067-c963ac57113e\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:42.739524 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:42.739477 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:43.106890 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:43.106807 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79bf796845-jkss4"] Nov 03 23:09:43.113253 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:09:43.113220 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73570236_6aa6_4aa6_8067_c963ac57113e.slice/crio-39a24c6e41351e9d88a91bc38248c0a9439ac3a28912516f3c6fc8fc5643ef72 WatchSource:0}: Error finding container 39a24c6e41351e9d88a91bc38248c0a9439ac3a28912516f3c6fc8fc5643ef72: Status 404 returned error can't find the container with id 39a24c6e41351e9d88a91bc38248c0a9439ac3a28912516f3c6fc8fc5643ef72 Nov 03 23:09:43.114619 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:43.114602 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:09:43.718889 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:43.718851 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" event={"ID":"73570236-6aa6-4aa6-8067-c963ac57113e","Type":"ContainerStarted","Data":"39a24c6e41351e9d88a91bc38248c0a9439ac3a28912516f3c6fc8fc5643ef72"} Nov 03 23:09:44.725859 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:44.725813 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" event={"ID":"73570236-6aa6-4aa6-8067-c963ac57113e","Type":"ContainerStarted","Data":"fc60ff651dbd0c0964a89dd6d0bef767687c25b3461a89d7471d7fec6b2947d1"} Nov 03 23:09:44.726282 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:44.725937 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:44.833188 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:44.833120 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" podStartSLOduration=1.828160515 podStartE2EDuration="2.833102137s" totalImagesPullingTime="1.004941622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:44.832519609 +0000 UTC m=+3704.244705625" watchObservedRunningTime="2026-11-03 23:09:44.833102137 +0000 UTC m=+3704.245288155" Nov 03 23:09:50.737034 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:50.737000 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79bf796845-jkss4" Nov 03 23:09:50.891527 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:50.891465 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq"] Nov 03 23:09:50.891766 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:50.891723 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" podUID="c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" containerName="maas-api" containerID="cri-o://a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9" gracePeriod=30 Nov 03 23:09:51.301903 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:51.301866 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" podUID="c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.29:8443/health\": dial tcp 10.134.0.29:8443: connect: connection refused" Nov 03 23:09:52.130973 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.130945 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 23:09:52.238822 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.238743 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-maas-api-tls\") pod \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " Nov 03 23:09:52.238976 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.238840 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-openshift-service-ca\") pod \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " Nov 03 23:09:52.238976 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.238915 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-kube-api-access-l2xc9\" (UniqueName: \"kubernetes.io/projected/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-kube-api-access-l2xc9\") pod \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " Nov 03 23:09:52.238976 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.238958 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-metrics-certs\") pod \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\" (UID: \"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a\") " Nov 03 23:09:52.239310 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.239281 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-openshift-service-ca" pod "c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" (UID: "c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:52.241953 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.241925 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-kube-api-access-l2xc9" pod "c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" (UID: "c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a"). InnerVolumeSpecName "kube-api-access-l2xc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:52.243219 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.243180 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-maas-api-tls" pod "c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" (UID: "c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:52.243403 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.243376 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-metrics-certs" pod "c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" (UID: "c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:52.340664 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.340635 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-openshift-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:52.340664 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.340661 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2xc9\" (UniqueName: \"kubernetes.io/projected/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-kube-api-access-l2xc9\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:52.340664 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.340672 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-metrics-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:52.340890 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.340681 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a-maas-api-tls\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:52.766842 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.766804 6234 generic.go:356] "Generic (PLEG): container finished" podID="c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" containerID="a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9" exitCode=0 Nov 03 23:09:52.767007 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.766864 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" Nov 03 23:09:52.767007 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.766903 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" event={"ID":"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a","Type":"ContainerDied","Data":"a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9"} Nov 03 23:09:52.767007 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.766947 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq" event={"ID":"c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a","Type":"ContainerDied","Data":"837f9c42fa5ca29c6974dbaa8498ea9b2b829ab58a3d4ca168a091a7e6e61914"} Nov 03 23:09:52.767007 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.766976 6234 scope.go:122] "RemoveContainer" containerID="a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9" Nov 03 23:09:52.778956 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.778832 6234 scope.go:122] "RemoveContainer" containerID="a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9" Nov 03 23:09:52.779307 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:09:52.779278 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9\": container with ID starting with a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9 not found: ID does not exist" containerID="a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9" Nov 03 23:09:52.779377 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.779324 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9"} err="failed to get container status \"a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9\": rpc error: code = NotFound desc = could not find container \"a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9\": container with ID starting with a0c4700f9ddc6809e9d97f9471eb2aaa8f8437a44e19dc29166428f1cc92fbd9 not found: ID does not exist" Nov 03 23:09:52.919960 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.919926 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq"] Nov 03 23:09:52.966676 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:52.966593 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-dkjrq"] Nov 03 23:09:54.704336 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:54.704303 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" path="/var/lib/kubelet/pods/c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a/volumes" Nov 03 23:09:57.265654 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.265621 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75"] Nov 03 23:09:57.266057 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.266042 6234 cpu_manager.go:414] "containerMap: removing container" podUID="c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" containerName="maas-api" Nov 03 23:09:57.266098 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.266060 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" containerName="maas-api" Nov 03 23:09:57.266153 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.266144 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c8c1a2-4c41-4c58-a7b8-00144d3ae99a" containerName="maas-api" Nov 03 23:09:57.270646 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.270625 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:09:57.322987 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.322961 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75"] Nov 03 23:09:57.367788 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.367757 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f18d5a31-dockercfg-cw487\"" Nov 03 23:09:57.368042 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.367797 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f18d5a31\"" Nov 03 23:09:57.390690 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.390655 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcbl6\" (UniqueName: \"kubernetes.io/projected/3bf28836-f700-405a-8fc4-a542d6227082-kube-api-access-bcbl6\") pod \"payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75\" (UID: \"3bf28836-f700-405a-8fc4-a542d6227082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:09:57.390902 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.390702 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf28836-f700-405a-8fc4-a542d6227082-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75\" (UID: \"3bf28836-f700-405a-8fc4-a542d6227082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:09:57.415975 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.415946 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5"] Nov 03 23:09:57.419472 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.419450 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:09:57.469866 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.469839 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5"] Nov 03 23:09:57.491823 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.491790 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxvdq\" (UniqueName: \"kubernetes.io/projected/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-kube-api-access-jxvdq\") pod \"payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5\" (UID: \"4dbd66ac-a209-4de2-8b32-3f7a110f7acc\") " pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:09:57.491942 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.491899 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcbl6\" (UniqueName: \"kubernetes.io/projected/3bf28836-f700-405a-8fc4-a542d6227082-kube-api-access-bcbl6\") pod \"payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75\" (UID: \"3bf28836-f700-405a-8fc4-a542d6227082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:09:57.491942 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.491927 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-plugins-config-volume\") pod \"payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5\" (UID: \"4dbd66ac-a209-4de2-8b32-3f7a110f7acc\") " pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:09:57.492034 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.491978 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf28836-f700-405a-8fc4-a542d6227082-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75\" (UID: \"3bf28836-f700-405a-8fc4-a542d6227082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:09:57.492667 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.492647 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf28836-f700-405a-8fc4-a542d6227082-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75\" (UID: \"3bf28836-f700-405a-8fc4-a542d6227082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:09:57.545019 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.544958 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcbl6\" (UniqueName: \"kubernetes.io/projected/3bf28836-f700-405a-8fc4-a542d6227082-kube-api-access-bcbl6\") pod \"payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75\" (UID: \"3bf28836-f700-405a-8fc4-a542d6227082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:09:57.582914 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.582887 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:09:57.593165 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.593139 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-plugins-config-volume\") pod \"payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5\" (UID: \"4dbd66ac-a209-4de2-8b32-3f7a110f7acc\") " pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:09:57.593287 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.593269 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvdq\" (UniqueName: \"kubernetes.io/projected/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-kube-api-access-jxvdq\") pod \"payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5\" (UID: \"4dbd66ac-a209-4de2-8b32-3f7a110f7acc\") " pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:09:57.594083 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.594062 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-plugins-config-volume\") pod \"payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5\" (UID: \"4dbd66ac-a209-4de2-8b32-3f7a110f7acc\") " pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:09:57.649344 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.649309 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxvdq\" (UniqueName: \"kubernetes.io/projected/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-kube-api-access-jxvdq\") pod \"payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5\" (UID: \"4dbd66ac-a209-4de2-8b32-3f7a110f7acc\") " pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:09:57.731643 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.731606 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:09:57.927443 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.927364 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt"] Nov 03 23:09:57.932073 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.932050 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:57.949150 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.949123 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75"] Nov 03 23:09:57.956116 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:09:57.956093 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf28836_f700_405a_8fc4_a542d6227082.slice/crio-f6214833271cec03ffcec5171d2d14b4d4d520c89b3d58033b52f2657b64ad60 WatchSource:0}: Error finding container f6214833271cec03ffcec5171d2d14b4d4d520c89b3d58033b52f2657b64ad60: Status 404 returned error can't find the container with id f6214833271cec03ffcec5171d2d14b4d4d520c89b3d58033b52f2657b64ad60 Nov 03 23:09:57.986263 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.986236 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt"] Nov 03 23:09:57.986382 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.986362 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f18d5a31-serving-cert\"" Nov 03 23:09:57.997149 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.997115 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-openshift-service-ca\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:57.997266 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.997171 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-metrics-certs\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:57.997338 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.997268 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-maas-api-tls\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:57.997417 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:57.997394 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcfm\" (UniqueName: \"kubernetes.io/projected/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-kube-api-access-sdcfm\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.098390 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.098358 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcfm\" (UniqueName: \"kubernetes.io/projected/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-kube-api-access-sdcfm\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.098599 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.098448 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-openshift-service-ca\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.098599 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.098480 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-metrics-certs\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.098747 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.098619 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-maas-api-tls\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.098797 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:09:58.098751 6234 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-serving-cert: secret "maas-api-e2e-ait-f18d5a31-serving-cert" not found Nov 03 23:09:58.098830 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:09:58.098816 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-maas-api-tls podName:d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5 nodeName:}" failed. No retries permitted until 2026-11-03 23:09:58.598798679 +0000 UTC m=+3718.010984678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-maas-api-tls") pod "maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" (UID: "d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5") : secret "maas-api-e2e-ait-f18d5a31-serving-cert" not found Nov 03 23:09:58.099246 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.099220 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-openshift-service-ca\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.100882 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.100861 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-metrics-certs\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.104722 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.104700 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5"] Nov 03 23:09:58.110251 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:09:58.110223 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbd66ac_a209_4de2_8b32_3f7a110f7acc.slice/crio-d9c5c004c968a5a932dab294139e66d4138048fa20555e09919e64bc3969998a WatchSource:0}: Error finding container d9c5c004c968a5a932dab294139e66d4138048fa20555e09919e64bc3969998a: Status 404 returned error can't find the container with id d9c5c004c968a5a932dab294139e66d4138048fa20555e09919e64bc3969998a Nov 03 23:09:58.152393 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.152365 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcfm\" (UniqueName: \"kubernetes.io/projected/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-kube-api-access-sdcfm\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.604362 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.604329 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-maas-api-tls\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.606842 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.606816 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-maas-api-tls\") pod \"maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.800528 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.800476 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" event={"ID":"3bf28836-f700-405a-8fc4-a542d6227082","Type":"ContainerStarted","Data":"6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a"} Nov 03 23:09:58.800707 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.800550 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" event={"ID":"3bf28836-f700-405a-8fc4-a542d6227082","Type":"ContainerStarted","Data":"f6214833271cec03ffcec5171d2d14b4d4d520c89b3d58033b52f2657b64ad60"} Nov 03 23:09:58.800707 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.800579 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:09:58.802088 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.802066 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" event={"ID":"4dbd66ac-a209-4de2-8b32-3f7a110f7acc","Type":"ContainerStarted","Data":"c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c"} Nov 03 23:09:58.802237 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.802099 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" event={"ID":"4dbd66ac-a209-4de2-8b32-3f7a110f7acc","Type":"ContainerStarted","Data":"d9c5c004c968a5a932dab294139e66d4138048fa20555e09919e64bc3969998a"} Nov 03 23:09:58.802237 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.802130 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:09:58.846480 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.846452 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:09:58.908316 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:58.908246 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" podStartSLOduration=1.908228445 podStartE2EDuration="1.908228445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:58.907087919 +0000 UTC m=+3718.319273937" watchObservedRunningTime="2026-11-03 23:09:58.908228445 +0000 UTC m=+3718.320414478" Nov 03 23:09:58.995222 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:09:58.995182 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:09:59.014568 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:59.014512 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" podStartSLOduration=2.014475606 podStartE2EDuration="2.014475606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:59.012682452 +0000 UTC m=+3718.424868471" watchObservedRunningTime="2026-11-03 23:09:59.014475606 +0000 UTC m=+3718.426661624" Nov 03 23:09:59.271993 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:59.271959 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt"] Nov 03 23:09:59.278316 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:09:59.278279 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fa2d03_5f08_43ca_8fa3_f3ea1f7a6cd5.slice/crio-a6f349329852598762120212fd65c445c7158a25d8cb8ff3c800be987248d0f8 WatchSource:0}: Error finding container a6f349329852598762120212fd65c445c7158a25d8cb8ff3c800be987248d0f8: Status 404 returned error can't find the container with id a6f349329852598762120212fd65c445c7158a25d8cb8ff3c800be987248d0f8 Nov 03 23:09:59.807893 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:09:59.807860 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" event={"ID":"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5","Type":"ContainerStarted","Data":"a6f349329852598762120212fd65c445c7158a25d8cb8ff3c800be987248d0f8"} Nov 03 23:10:00.814914 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:00.814880 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" event={"ID":"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5","Type":"ContainerStarted","Data":"da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de"} Nov 03 23:10:00.815292 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:00.815013 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:10:00.923321 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:00.923268 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" podStartSLOduration=3.009604304 podStartE2EDuration="3.923255243s" totalImagesPullingTime="913.650939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:10:00.922370608 +0000 UTC m=+3720.334556625" watchObservedRunningTime="2026-11-03 23:10:00.923255243 +0000 UTC m=+3720.335441260" Nov 03 23:10:06.462181 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:06.462138 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:10:06.825721 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:06.825649 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:10:09.811265 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:09.811234 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:10:09.811701 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:09.811298 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:10:28.998593 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:10:28.998549 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:10:41.544811 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:41.544773 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5"] Nov 03 23:10:41.651798 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:41.651765 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75"] Nov 03 23:10:46.545815 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.545747 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" podUID="4dbd66ac-a209-4de2-8b32-3f7a110f7acc" containerName="payload-processing" containerID="cri-o://c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c" gracePeriod=25 Nov 03 23:10:46.652074 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.652014 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" podUID="3bf28836-f700-405a-8fc4-a542d6227082" containerName="payload-pre-processing" containerID="cri-o://6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a" gracePeriod=25 Nov 03 23:10:46.798379 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.798322 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:10:46.895971 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.895944 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-plugins-config-volume\") pod \"4dbd66ac-a209-4de2-8b32-3f7a110f7acc\" (UID: \"4dbd66ac-a209-4de2-8b32-3f7a110f7acc\") " Nov 03 23:10:46.896098 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.896010 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-kube-api-access-jxvdq\" (UniqueName: \"kubernetes.io/projected/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-kube-api-access-jxvdq\") pod \"4dbd66ac-a209-4de2-8b32-3f7a110f7acc\" (UID: \"4dbd66ac-a209-4de2-8b32-3f7a110f7acc\") " Nov 03 23:10:46.896328 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.896306 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-plugins-config-volume" pod "4dbd66ac-a209-4de2-8b32-3f7a110f7acc" (UID: "4dbd66ac-a209-4de2-8b32-3f7a110f7acc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:46.897985 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.897961 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-kube-api-access-jxvdq" pod "4dbd66ac-a209-4de2-8b32-3f7a110f7acc" (UID: "4dbd66ac-a209-4de2-8b32-3f7a110f7acc"). InnerVolumeSpecName "kube-api-access-jxvdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:46.902699 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.902678 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:10:46.996895 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.996868 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf28836-f700-405a-8fc4-a542d6227082-kube-api-access-bcbl6\" (UniqueName: \"kubernetes.io/projected/3bf28836-f700-405a-8fc4-a542d6227082-kube-api-access-bcbl6\") pod \"3bf28836-f700-405a-8fc4-a542d6227082\" (UID: \"3bf28836-f700-405a-8fc4-a542d6227082\") " Nov 03 23:10:46.997064 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.996984 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf28836-f700-405a-8fc4-a542d6227082-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf28836-f700-405a-8fc4-a542d6227082-plugins-config-volume\") pod \"3bf28836-f700-405a-8fc4-a542d6227082\" (UID: \"3bf28836-f700-405a-8fc4-a542d6227082\") " Nov 03 23:10:46.997324 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.997304 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxvdq\" (UniqueName: \"kubernetes.io/projected/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-kube-api-access-jxvdq\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:46.997383 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.997330 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbd66ac-a209-4de2-8b32-3f7a110f7acc-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:46.997421 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.997379 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf28836-f700-405a-8fc4-a542d6227082-plugins-config-volume" pod "3bf28836-f700-405a-8fc4-a542d6227082" (UID: "3bf28836-f700-405a-8fc4-a542d6227082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:46.998971 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:46.998949 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf28836-f700-405a-8fc4-a542d6227082-kube-api-access-bcbl6" pod "3bf28836-f700-405a-8fc4-a542d6227082" (UID: "3bf28836-f700-405a-8fc4-a542d6227082"). InnerVolumeSpecName "kube-api-access-bcbl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:47.031188 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.031156 6234 generic.go:356] "Generic (PLEG): container finished" podID="4dbd66ac-a209-4de2-8b32-3f7a110f7acc" containerID="c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c" exitCode=0 Nov 03 23:10:47.031359 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.031200 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" Nov 03 23:10:47.031359 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.031238 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" event={"ID":"4dbd66ac-a209-4de2-8b32-3f7a110f7acc","Type":"ContainerDied","Data":"c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c"} Nov 03 23:10:47.031359 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.031291 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5" event={"ID":"4dbd66ac-a209-4de2-8b32-3f7a110f7acc","Type":"ContainerDied","Data":"d9c5c004c968a5a932dab294139e66d4138048fa20555e09919e64bc3969998a"} Nov 03 23:10:47.031359 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.031318 6234 scope.go:122] "RemoveContainer" containerID="c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c" Nov 03 23:10:47.032910 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.032881 6234 generic.go:356] "Generic (PLEG): container finished" podID="3bf28836-f700-405a-8fc4-a542d6227082" containerID="6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a" exitCode=0 Nov 03 23:10:47.033010 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.032943 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" Nov 03 23:10:47.033460 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.032947 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" event={"ID":"3bf28836-f700-405a-8fc4-a542d6227082","Type":"ContainerDied","Data":"6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a"} Nov 03 23:10:47.033567 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.033511 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75" event={"ID":"3bf28836-f700-405a-8fc4-a542d6227082","Type":"ContainerDied","Data":"f6214833271cec03ffcec5171d2d14b4d4d520c89b3d58033b52f2657b64ad60"} Nov 03 23:10:47.044051 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.044007 6234 scope.go:122] "RemoveContainer" containerID="c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c" Nov 03 23:10:47.044323 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:10:47.044300 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c\": container with ID starting with c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c not found: ID does not exist" containerID="c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c" Nov 03 23:10:47.044419 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.044335 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c"} err="failed to get container status \"c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c\": rpc error: code = NotFound desc = could not find container \"c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c\": container with ID starting with c02033b5978a831740c462633d0a0faca7a522f23e0b05330c1b52da326aaf1c not found: ID does not exist" Nov 03 23:10:47.044419 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.044357 6234 scope.go:122] "RemoveContainer" containerID="6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a" Nov 03 23:10:47.053030 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.053010 6234 scope.go:122] "RemoveContainer" containerID="6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a" Nov 03 23:10:47.053290 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:10:47.053271 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a\": container with ID starting with 6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a not found: ID does not exist" containerID="6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a" Nov 03 23:10:47.053348 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.053301 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a"} err="failed to get container status \"6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a\": rpc error: code = NotFound desc = could not find container \"6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a\": container with ID starting with 6da970f5c47be5c86a72065120b40a91081f0a176cf457c93b963cc90f52307a not found: ID does not exist" Nov 03 23:10:47.098029 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.098003 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf28836-f700-405a-8fc4-a542d6227082-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:47.098029 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.098026 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcbl6\" (UniqueName: \"kubernetes.io/projected/3bf28836-f700-405a-8fc4-a542d6227082-kube-api-access-bcbl6\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:47.185686 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.185657 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75"] Nov 03 23:10:47.189830 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.189806 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f18d5a31-9c95549d7-npc75"] Nov 03 23:10:47.291262 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.291225 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5"] Nov 03 23:10:47.298119 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.298076 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f18d5a31-54c5bf94dd-fb2z5"] Nov 03 23:10:47.686696 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.686656 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt"] Nov 03 23:10:47.687169 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.686950 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" podUID="d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5" containerName="maas-api" containerID="cri-o://da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de" gracePeriod=30 Nov 03 23:10:47.944566 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:47.944458 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:10:48.040145 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.040114 6234 generic.go:356] "Generic (PLEG): container finished" podID="d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5" containerID="da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de" exitCode=0 Nov 03 23:10:48.040332 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.040156 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" Nov 03 23:10:48.040332 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.040199 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" event={"ID":"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5","Type":"ContainerDied","Data":"da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de"} Nov 03 23:10:48.040332 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.040256 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt" event={"ID":"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5","Type":"ContainerDied","Data":"a6f349329852598762120212fd65c445c7158a25d8cb8ff3c800be987248d0f8"} Nov 03 23:10:48.040332 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.040285 6234 scope.go:122] "RemoveContainer" containerID="da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de" Nov 03 23:10:48.048575 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.048552 6234 scope.go:122] "RemoveContainer" containerID="da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de" Nov 03 23:10:48.048859 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:10:48.048843 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de\": container with ID starting with da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de not found: ID does not exist" containerID="da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de" Nov 03 23:10:48.048920 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.048870 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de"} err="failed to get container status \"da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de\": rpc error: code = NotFound desc = could not find container \"da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de\": container with ID starting with da11f700c52822e891c5df5eafb0a1835df5a4684a70238a33c899560a84e6de not found: ID does not exist" Nov 03 23:10:48.111725 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.111655 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-openshift-service-ca\") pod \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " Nov 03 23:10:48.111872 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.111746 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-metrics-certs\") pod \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " Nov 03 23:10:48.111872 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.111779 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-maas-api-tls\") pod \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " Nov 03 23:10:48.111872 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.111827 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-kube-api-access-sdcfm\" (UniqueName: \"kubernetes.io/projected/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-kube-api-access-sdcfm\") pod \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\" (UID: \"d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5\") " Nov 03 23:10:48.112097 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.112002 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-openshift-service-ca" pod "d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5" (UID: "d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:48.112199 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.112179 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-openshift-service-ca\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:48.113886 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.113864 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-maas-api-tls" pod "d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5" (UID: "d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:48.113993 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.113926 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-kube-api-access-sdcfm" pod "d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5" (UID: "d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5"). InnerVolumeSpecName "kube-api-access-sdcfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:48.113993 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.113938 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-metrics-certs" pod "d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5" (UID: "d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:48.212990 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.212956 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdcfm\" (UniqueName: \"kubernetes.io/projected/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-kube-api-access-sdcfm\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:48.212990 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.212988 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-metrics-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:48.212990 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.212998 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5-maas-api-tls\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:48.452604 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.452571 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt"] Nov 03 23:10:48.499444 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.499414 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f18d5a31-77d7cd798b-cmzxt"] Nov 03 23:10:48.703696 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.703610 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf28836-f700-405a-8fc4-a542d6227082" path="/var/lib/kubelet/pods/3bf28836-f700-405a-8fc4-a542d6227082/volumes" Nov 03 23:10:48.704189 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.704167 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dbd66ac-a209-4de2-8b32-3f7a110f7acc" path="/var/lib/kubelet/pods/4dbd66ac-a209-4de2-8b32-3f7a110f7acc/volumes" Nov 03 23:10:48.704626 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:48.704606 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5" path="/var/lib/kubelet/pods/d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5/volumes" Nov 03 23:10:52.470932 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:10:52.470621 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:10:58.996486 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:10:58.996446 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:11:22.960106 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.960028 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v"] Nov 03 23:11:22.960601 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.960430 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5" containerName="maas-api" Nov 03 23:11:22.960601 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.960441 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5" containerName="maas-api" Nov 03 23:11:22.960601 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.960462 6234 cpu_manager.go:414] "containerMap: removing container" podUID="4dbd66ac-a209-4de2-8b32-3f7a110f7acc" containerName="payload-processing" Nov 03 23:11:22.960601 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.960470 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dbd66ac-a209-4de2-8b32-3f7a110f7acc" containerName="payload-processing" Nov 03 23:11:22.960601 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.960481 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3bf28836-f700-405a-8fc4-a542d6227082" containerName="payload-pre-processing" Nov 03 23:11:22.960601 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.960486 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf28836-f700-405a-8fc4-a542d6227082" containerName="payload-pre-processing" Nov 03 23:11:22.960601 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.960581 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6fa2d03-5f08-43ca-8fa3-f3ea1f7a6cd5" containerName="maas-api" Nov 03 23:11:22.960601 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.960592 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dbd66ac-a209-4de2-8b32-3f7a110f7acc" containerName="payload-processing" Nov 03 23:11:22.960601 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.960599 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf28836-f700-405a-8fc4-a542d6227082" containerName="payload-pre-processing" Nov 03 23:11:22.963354 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:22.963312 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.044814 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.044780 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-podinfo\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.044986 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.044832 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-credential-socket\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.044986 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.044936 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istiod-ca-cert\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.044986 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.044961 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjl7k\" (UniqueName: \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-kube-api-access-jjl7k\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.045105 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.045003 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-certs\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.045105 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.045054 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-socket\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.045197 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.045112 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-data\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.045197 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.045134 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-envoy\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.045197 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.045159 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-token\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.059160 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.059130 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v"] Nov 03 23:11:23.070593 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.070567 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-26c7c9-openshift-default-dockercfg-4ml86\"" Nov 03 23:11:23.145971 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.145938 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istiod-ca-cert\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.145971 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.145980 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjl7k\" (UniqueName: \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-kube-api-access-jjl7k\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146262 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146013 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-certs\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146262 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146035 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-socket\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146262 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146072 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-data\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146262 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146093 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-envoy\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146262 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146127 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-token\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146262 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146188 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-podinfo\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146262 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146240 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-credential-socket\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146695 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146474 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-certs\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146695 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146569 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-data\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146695 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146617 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-credential-socket\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146838 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146720 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-socket\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.146893 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.146868 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istiod-ca-cert\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.149322 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.149290 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-podinfo\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.149998 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.149976 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-envoy\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.199600 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.199574 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-token\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.199888 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.199870 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjl7k\" (UniqueName: \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-kube-api-access-jjl7k\") pod \"e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.276930 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.276858 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:23.705394 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.705363 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v"] Nov 03 23:11:23.711838 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:11:23.711806 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0ed46b_9b77_41a1_b20c_8d1597b62cb2.slice/crio-106e23fcda6a375a74800f3cf796918379657ce0f13be6e30ee8421f652f5b0a WatchSource:0}: Error finding container 106e23fcda6a375a74800f3cf796918379657ce0f13be6e30ee8421f652f5b0a: Status 404 returned error can't find the container with id 106e23fcda6a375a74800f3cf796918379657ce0f13be6e30ee8421f652f5b0a Nov 03 23:11:23.714001 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.713966 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:11:23.714106 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.714036 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:11:23.714106 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:23.714065 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:11:24.212709 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:24.212672 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" event={"ID":"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2","Type":"ContainerStarted","Data":"839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f"} Nov 03 23:11:24.213179 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:24.212725 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" event={"ID":"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2","Type":"ContainerStarted","Data":"106e23fcda6a375a74800f3cf796918379657ce0f13be6e30ee8421f652f5b0a"} Nov 03 23:11:24.277831 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:24.277799 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:24.282978 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:24.282952 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:24.322645 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:24.322595 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" podStartSLOduration=2.322582412 podStartE2EDuration="2.322582412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:24.321650837 +0000 UTC m=+3803.733836855" watchObservedRunningTime="2026-11-03 23:11:24.322582412 +0000 UTC m=+3803.734768670" Nov 03 23:11:25.218272 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:25.218245 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:25.219353 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:25.219336 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:28.995887 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:11:28.995844 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:11:31.105263 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.105228 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794"] Nov 03 23:11:31.109742 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.109721 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:11:31.126704 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.126673 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0832a4-9817-4359-ab38-ed66c9edb28c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794\" (UID: \"2f0832a4-9817-4359-ab38-ed66c9edb28c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:11:31.126844 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.126722 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwdz\" (UniqueName: \"kubernetes.io/projected/2f0832a4-9817-4359-ab38-ed66c9edb28c-kube-api-access-lgwdz\") pod \"payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794\" (UID: \"2f0832a4-9817-4359-ab38-ed66c9edb28c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:11:31.160056 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.160030 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794"] Nov 03 23:11:31.205308 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.205281 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-26c7c9\"" Nov 03 23:11:31.205308 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.205287 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-26c7c9-dockercfg-qmvmj\"" Nov 03 23:11:31.228261 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.228232 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0832a4-9817-4359-ab38-ed66c9edb28c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794\" (UID: \"2f0832a4-9817-4359-ab38-ed66c9edb28c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:11:31.228400 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.228274 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwdz\" (UniqueName: \"kubernetes.io/projected/2f0832a4-9817-4359-ab38-ed66c9edb28c-kube-api-access-lgwdz\") pod \"payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794\" (UID: \"2f0832a4-9817-4359-ab38-ed66c9edb28c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:11:31.228942 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.228920 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0832a4-9817-4359-ab38-ed66c9edb28c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794\" (UID: \"2f0832a4-9817-4359-ab38-ed66c9edb28c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:11:31.285448 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.285413 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwdz\" (UniqueName: \"kubernetes.io/projected/2f0832a4-9817-4359-ab38-ed66c9edb28c-kube-api-access-lgwdz\") pod \"payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794\" (UID: \"2f0832a4-9817-4359-ab38-ed66c9edb28c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:11:31.427487 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.427400 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:11:31.792007 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:31.791971 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794"] Nov 03 23:11:31.798625 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:11:31.798597 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0832a4_9817_4359_ab38_ed66c9edb28c.slice/crio-55265bf877e32aa07c68f3cd0a6ae524290b7697291d389aebfecbdf0780357a WatchSource:0}: Error finding container 55265bf877e32aa07c68f3cd0a6ae524290b7697291d389aebfecbdf0780357a: Status 404 returned error can't find the container with id 55265bf877e32aa07c68f3cd0a6ae524290b7697291d389aebfecbdf0780357a Nov 03 23:11:32.256068 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:32.256031 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" event={"ID":"2f0832a4-9817-4359-ab38-ed66c9edb28c","Type":"ContainerStarted","Data":"48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0"} Nov 03 23:11:32.256549 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:32.256080 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" event={"ID":"2f0832a4-9817-4359-ab38-ed66c9edb28c","Type":"ContainerStarted","Data":"55265bf877e32aa07c68f3cd0a6ae524290b7697291d389aebfecbdf0780357a"} Nov 03 23:11:32.256549 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:32.256227 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:11:32.363800 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:32.363752 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" podStartSLOduration=1.363736383 podStartE2EDuration="1.363736383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:32.362205454 +0000 UTC m=+3811.774391471" watchObservedRunningTime="2026-11-03 23:11:32.363736383 +0000 UTC m=+3811.775922432" Nov 03 23:11:42.063434 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:42.063397 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:11:43.266662 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:43.266634 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:11:51.344656 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:51.344619 6234 generic.go:356] "Generic (PLEG): container finished" podID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerID="839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f" exitCode=0 Nov 03 23:11:51.345111 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:51.344662 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" event={"ID":"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2","Type":"ContainerDied","Data":"839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f"} Nov 03 23:11:51.345111 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:51.344999 6234 scope.go:122] "RemoveContainer" containerID="839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f" Nov 03 23:11:51.346217 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:51.346184 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:11:51.346297 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:51.346259 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:11:51.346334 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:51.346298 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:11:52.369882 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:52.369847 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" event={"ID":"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2","Type":"ContainerStarted","Data":"9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe"} Nov 03 23:11:53.277475 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:53.277443 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:53.278662 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:53.278632 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 23:11:53.278812 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:53.278680 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 23:11:54.277993 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:54.277953 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 23:11:54.278602 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:54.278013 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 23:11:55.218635 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:55.218599 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:55.277715 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:55.277681 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 23:11:55.277900 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:55.277736 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 23:11:56.277614 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:56.277579 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 23:11:56.278033 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:56.277630 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 23:11:57.277478 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:57.277439 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 23:11:57.277732 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:57.277530 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 23:11:58.281861 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:58.281830 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:58.282823 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:11:58.282800 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:11:58.995271 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:11:58.995235 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:12:28.996555 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:12:28.996514 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:12:58.996058 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:12:58.996013 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:13:23.994144 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:23.994102 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794"] Nov 03 23:13:28.994508 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:28.994433 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" podUID="2f0832a4-9817-4359-ab38-ed66c9edb28c" containerName="payload-processing" containerID="cri-o://48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0" gracePeriod=25 Nov 03 23:13:28.995622 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:13:28.995593 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:13:29.240889 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.240866 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:13:29.330184 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.330112 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0832a4-9817-4359-ab38-ed66c9edb28c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0832a4-9817-4359-ab38-ed66c9edb28c-plugins-config-volume\") pod \"2f0832a4-9817-4359-ab38-ed66c9edb28c\" (UID: \"2f0832a4-9817-4359-ab38-ed66c9edb28c\") " Nov 03 23:13:29.330316 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.330193 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0832a4-9817-4359-ab38-ed66c9edb28c-kube-api-access-lgwdz\" (UniqueName: \"kubernetes.io/projected/2f0832a4-9817-4359-ab38-ed66c9edb28c-kube-api-access-lgwdz\") pod \"2f0832a4-9817-4359-ab38-ed66c9edb28c\" (UID: \"2f0832a4-9817-4359-ab38-ed66c9edb28c\") " Nov 03 23:13:29.330481 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.330458 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0832a4-9817-4359-ab38-ed66c9edb28c-plugins-config-volume" pod "2f0832a4-9817-4359-ab38-ed66c9edb28c" (UID: "2f0832a4-9817-4359-ab38-ed66c9edb28c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:13:29.330635 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.330617 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0832a4-9817-4359-ab38-ed66c9edb28c-plugins-config-volume\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:29.332322 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.332302 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0832a4-9817-4359-ab38-ed66c9edb28c-kube-api-access-lgwdz" pod "2f0832a4-9817-4359-ab38-ed66c9edb28c" (UID: "2f0832a4-9817-4359-ab38-ed66c9edb28c"). InnerVolumeSpecName "kube-api-access-lgwdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:29.431759 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.431729 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgwdz\" (UniqueName: \"kubernetes.io/projected/2f0832a4-9817-4359-ab38-ed66c9edb28c-kube-api-access-lgwdz\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:29.826663 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.826627 6234 generic.go:356] "Generic (PLEG): container finished" podID="2f0832a4-9817-4359-ab38-ed66c9edb28c" containerID="48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0" exitCode=0 Nov 03 23:13:29.826843 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.826676 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" Nov 03 23:13:29.826843 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.826711 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" event={"ID":"2f0832a4-9817-4359-ab38-ed66c9edb28c","Type":"ContainerDied","Data":"48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0"} Nov 03 23:13:29.826843 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.826765 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794" event={"ID":"2f0832a4-9817-4359-ab38-ed66c9edb28c","Type":"ContainerDied","Data":"55265bf877e32aa07c68f3cd0a6ae524290b7697291d389aebfecbdf0780357a"} Nov 03 23:13:29.826843 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.826786 6234 scope.go:122] "RemoveContainer" containerID="48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0" Nov 03 23:13:29.835634 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.835616 6234 scope.go:122] "RemoveContainer" containerID="48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0" Nov 03 23:13:29.835906 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:13:29.835887 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0\": container with ID starting with 48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0 not found: ID does not exist" containerID="48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0" Nov 03 23:13:29.835994 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.835913 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0"} err="failed to get container status \"48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0\": rpc error: code = NotFound desc = could not find container \"48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0\": container with ID starting with 48a0bbb7ac63cf76f828f643819dd52e295c8a043400fbeb4b8b09b4868bd2c0 not found: ID does not exist" Nov 03 23:13:29.978863 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.978833 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794"] Nov 03 23:13:29.982505 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:29.982465 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-26c7c9-7d57f7454d-49794"] Nov 03 23:13:30.703531 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:30.703481 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0832a4-9817-4359-ab38-ed66c9edb28c" path="/var/lib/kubelet/pods/2f0832a4-9817-4359-ab38-ed66c9edb28c/volumes" Nov 03 23:13:37.669255 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:37.669224 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:13:41.548152 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:41.548118 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v"] Nov 03 23:13:41.548612 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:41.548328 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" containerID="cri-o://9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe" gracePeriod=30 Nov 03 23:13:43.285062 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:43.285025 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:13:43.285440 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:43.285107 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:13:46.793529 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.793482 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:13:46.813725 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.813675 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-kube-api-access-jjl7k\" (UniqueName: \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-kube-api-access-jjl7k\") pod \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " Nov 03 23:13:46.813805 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.813752 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-podinfo\") pod \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " Nov 03 23:13:46.813805 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.813796 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-data\") pod \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " Nov 03 23:13:46.813906 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.813822 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-credential-socket\") pod \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " Nov 03 23:13:46.813906 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.813838 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-token\" (UniqueName: \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-token\") pod \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " Nov 03 23:13:46.813906 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.813865 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-envoy\") pod \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " Nov 03 23:13:46.814209 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814081 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-certs\") pod \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " Nov 03 23:13:46.814209 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814113 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-data" pod "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" (UID: "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:13:46.814209 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814137 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istiod-ca-cert\") pod \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " Nov 03 23:13:46.814209 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814136 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-credential-socket" pod "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" (UID: "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:13:46.814209 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814192 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-socket\") pod \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\" (UID: \"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2\") " Nov 03 23:13:46.814591 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814292 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-certs" pod "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" (UID: "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:13:46.814591 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814456 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istiod-ca-cert" pod "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" (UID: "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:13:46.814591 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814575 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-socket" pod "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" (UID: "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:13:46.814754 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814735 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-data\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:46.814791 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814762 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-credential-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:46.814791 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814778 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-certs\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:46.814854 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814793 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istiod-ca-cert\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:46.814854 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.814807 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-workload-socket\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:46.816074 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.816047 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-token" pod "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" (UID: "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:46.816154 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.816090 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-envoy" pod "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" (UID: "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:13:46.816336 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.816313 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-kube-api-access-jjl7k" pod "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" (UID: "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2"). InnerVolumeSpecName "kube-api-access-jjl7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:46.816336 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.816323 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-podinfo" pod "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" (UID: "cc0ed46b-9b77-41a1-b20c-8d1597b62cb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:13:46.910933 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.910898 6234 generic.go:356] "Generic (PLEG): container finished" podID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerID="9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe" exitCode=0 Nov 03 23:13:46.911093 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.910948 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" Nov 03 23:13:46.911093 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.910985 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" event={"ID":"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2","Type":"ContainerDied","Data":"9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe"} Nov 03 23:13:46.911093 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.911041 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v" event={"ID":"cc0ed46b-9b77-41a1-b20c-8d1597b62cb2","Type":"ContainerDied","Data":"106e23fcda6a375a74800f3cf796918379657ce0f13be6e30ee8421f652f5b0a"} Nov 03 23:13:46.911093 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.911070 6234 scope.go:122] "RemoveContainer" containerID="9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe" Nov 03 23:13:46.915539 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.915516 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-envoy\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:46.915642 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.915544 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjl7k\" (UniqueName: \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-kube-api-access-jjl7k\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:46.915642 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.915562 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-podinfo\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:46.915642 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.915577 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2-istio-token\") on node \"ip-10-0-134-57.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:46.922866 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.922847 6234 scope.go:122] "RemoveContainer" containerID="839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f" Nov 03 23:13:46.930796 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.930778 6234 scope.go:122] "RemoveContainer" containerID="9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe" Nov 03 23:13:46.931071 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:13:46.931053 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe\": container with ID starting with 9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe not found: ID does not exist" containerID="9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe" Nov 03 23:13:46.931131 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.931080 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe"} err="failed to get container status \"9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe\": rpc error: code = NotFound desc = could not find container \"9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe\": container with ID starting with 9847eeefe7e895f5ec27c281249f0a5be5e5b8fef976ad52f6133acb81c0e4fe not found: ID does not exist" Nov 03 23:13:46.931131 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.931101 6234 scope.go:122] "RemoveContainer" containerID="839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f" Nov 03 23:13:46.931361 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:13:46.931337 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f\": container with ID starting with 839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f not found: ID does not exist" containerID="839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f" Nov 03 23:13:46.931415 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:46.931369 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f"} err="failed to get container status \"839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f\": rpc error: code = NotFound desc = could not find container \"839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f\": container with ID starting with 839b75f0a0bd487e408fa10ab1475ef9dc4ca6905840251aa68653155fa8f79f not found: ID does not exist" Nov 03 23:13:47.054536 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:47.054490 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v"] Nov 03 23:13:47.059939 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:47.059912 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-26c7c9-openshift-default-6fb55c9f65-zqm6v"] Nov 03 23:13:48.703364 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:48.703328 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" path="/var/lib/kubelet/pods/cc0ed46b-9b77-41a1-b20c-8d1597b62cb2/volumes" Nov 03 23:13:49.687229 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:13:49.687195 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:13:58.996354 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:13:58.996310 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:14:00.782060 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:14:00.782011 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:14:21.956442 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:14:21.956358 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:14:29.084842 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:14:29.084801 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:14:55.206825 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:14:55.206794 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-656fb666db-8fhmz_ef51a905-8e84-496a-8b7e-02cbdd3a2395/authorino/0.log" Nov 03 23:14:55.893466 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:14:55.893155 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-274zz"] Nov 03 23:14:58.995550 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:14:58.995514 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:15:09.560759 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:09.560725 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79bf796845-jkss4_73570236-6aa6-4aa6-8067-c963ac57113e/maas-api/0.log" Nov 03 23:15:12.543564 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:12.543527 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-656fb666db-8fhmz_ef51a905-8e84-496a-8b7e-02cbdd3a2395/authorino/0.log" Nov 03 23:15:14.053158 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:14.053129 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-274zz_f669b176-901b-4865-8735-b04ac3c7dff6/limitador/0.log" Nov 03 23:15:15.707646 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:15.707614 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dbbd9b584-9pvd9_77bbbc33-ee85-4061-922e-fbc1d70acc83/kube-auth-proxy/0.log" Nov 03 23:15:16.375154 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:16.375119 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-64665_7468570c-3547-4aa4-9d7e-c4f4aad4c395/istio-proxy/0.log" Nov 03 23:15:17.185903 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:17.185870 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-bqdwt_fce20530-f3fd-4d1a-9dcd-151248db648a/istio-proxy/0.log" Nov 03 23:15:18.231334 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:18.231302 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-w4b5m_7f3294b5-bf10-41a8-b126-6393399d4b40/istio-proxy/0.log" Nov 03 23:15:18.505440 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:18.505341 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-zh7jb_8e40f9f4-42b6-4c99-8b00-1ba515d7cf91/istio-proxy/0.log" Nov 03 23:15:20.282770 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:20.282736 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-nknft_98ac88a1-8c4e-4fac-b490-d39fe59a433a/main/0.log" Nov 03 23:15:20.558658 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:20.558590 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tt4nt_e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446/main/0.log" Nov 03 23:15:20.840921 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:20.840849 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-dql2n_435e9ee4-e39e-42f5-8a4b-e949f30b71a3/main/0.log" Nov 03 23:15:21.284469 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:21.284436 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf_608ba6ab-1275-43bf-9f86-ac190f8b8a99/main/0.log" Nov 03 23:15:21.741237 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:21.741206 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l_ac4a706f-973b-40e0-a54f-56da8b520e96/main/0.log" Nov 03 23:15:21.962721 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:21.962677 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8_48fcdc24-884a-482b-80cb-790ff881816d/main/0.log" Nov 03 23:15:22.892825 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:22.892787 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-656fb666db-8fhmz_ef51a905-8e84-496a-8b7e-02cbdd3a2395/authorino/0.log" Nov 03 23:15:29.005119 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:15:29.005083 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:15:34.997629 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:34.997595 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79bf796845-jkss4_73570236-6aa6-4aa6-8067-c963ac57113e/maas-api/0.log" Nov 03 23:15:37.120785 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:37.120744 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-656fb666db-8fhmz_ef51a905-8e84-496a-8b7e-02cbdd3a2395/authorino/0.log" Nov 03 23:15:38.514031 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:38.513999 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-274zz_f669b176-901b-4865-8735-b04ac3c7dff6/limitador/0.log" Nov 03 23:15:40.023360 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:40.023327 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dbbd9b584-9pvd9_77bbbc33-ee85-4061-922e-fbc1d70acc83/kube-auth-proxy/0.log" Nov 03 23:15:40.710809 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:40.710775 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-64665_7468570c-3547-4aa4-9d7e-c4f4aad4c395/istio-proxy/0.log" Nov 03 23:15:41.517898 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:41.517871 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-bqdwt_fce20530-f3fd-4d1a-9dcd-151248db648a/istio-proxy/0.log" Nov 03 23:15:42.543672 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:42.543637 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-w4b5m_7f3294b5-bf10-41a8-b126-6393399d4b40/istio-proxy/0.log" Nov 03 23:15:42.818649 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:42.818559 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-zh7jb_8e40f9f4-42b6-4c99-8b00-1ba515d7cf91/istio-proxy/0.log" Nov 03 23:15:44.396191 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:44.396156 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-nknft_98ac88a1-8c4e-4fac-b490-d39fe59a433a/main/0.log" Nov 03 23:15:44.675217 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:44.675129 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tt4nt_e54d8a0d-80b3-40a4-bc7a-cc3ad6e7a446/main/0.log" Nov 03 23:15:44.899321 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:44.899285 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-dql2n_435e9ee4-e39e-42f5-8a4b-e949f30b71a3/main/0.log" Nov 03 23:15:45.343940 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:45.343909 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wwzf_608ba6ab-1275-43bf-9f86-ac190f8b8a99/main/0.log" Nov 03 23:15:45.797293 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:45.797255 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-pzr4l_ac4a706f-973b-40e0-a54f-56da8b520e96/main/0.log" Nov 03 23:15:46.017396 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:15:46.017366 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-lfbn8_48fcdc24-884a-482b-80cb-790ff881816d/main/0.log" Nov 03 23:15:58.995789 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:15:58.995748 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:16:28.996911 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:16:28.996866 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:16:53.526676 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.526643 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9px9d/must-gather-s9knr"] Nov 03 23:16:53.528905 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.527029 6234 cpu_manager.go:414] "containerMap: removing container" podUID="2f0832a4-9817-4359-ab38-ed66c9edb28c" containerName="payload-processing" Nov 03 23:16:53.528905 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.527039 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0832a4-9817-4359-ab38-ed66c9edb28c" containerName="payload-processing" Nov 03 23:16:53.528905 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.527058 6234 cpu_manager.go:414] "containerMap: removing container" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" Nov 03 23:16:53.528905 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.527063 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" Nov 03 23:16:53.528905 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.527082 6234 cpu_manager.go:414] "containerMap: removing container" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" Nov 03 23:16:53.528905 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.527087 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" Nov 03 23:16:53.528905 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.527151 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" Nov 03 23:16:53.528905 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.527162 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0832a4-9817-4359-ab38-ed66c9edb28c" containerName="payload-processing" Nov 03 23:16:53.528905 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.527284 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0ed46b-9b77-41a1-b20c-8d1597b62cb2" containerName="istio-proxy" Nov 03 23:16:53.530413 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.530396 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9px9d/must-gather-s9knr" Nov 03 23:16:53.581793 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.581769 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9px9d\"/\"openshift-service-ca.crt\"" Nov 03 23:16:53.581947 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.581928 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9px9d\"/\"kube-root-ca.crt\"" Nov 03 23:16:53.582042 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.582031 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9px9d\"/\"default-dockercfg-546s9\"" Nov 03 23:16:53.593221 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.593193 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9px9d/must-gather-s9knr"] Nov 03 23:16:53.632672 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.632650 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w54dp\" (UniqueName: \"kubernetes.io/projected/0920caff-4f2a-42ef-a11a-d5243a11c5da-kube-api-access-w54dp\") pod \"must-gather-s9knr\" (UID: \"0920caff-4f2a-42ef-a11a-d5243a11c5da\") " pod="openshift-must-gather-9px9d/must-gather-s9knr" Nov 03 23:16:53.632798 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.632718 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0920caff-4f2a-42ef-a11a-d5243a11c5da-must-gather-output\") pod \"must-gather-s9knr\" (UID: \"0920caff-4f2a-42ef-a11a-d5243a11c5da\") " pod="openshift-must-gather-9px9d/must-gather-s9knr" Nov 03 23:16:53.733422 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.733391 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0920caff-4f2a-42ef-a11a-d5243a11c5da-must-gather-output\") pod \"must-gather-s9knr\" (UID: \"0920caff-4f2a-42ef-a11a-d5243a11c5da\") " pod="openshift-must-gather-9px9d/must-gather-s9knr" Nov 03 23:16:53.733619 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.733475 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w54dp\" (UniqueName: \"kubernetes.io/projected/0920caff-4f2a-42ef-a11a-d5243a11c5da-kube-api-access-w54dp\") pod \"must-gather-s9knr\" (UID: \"0920caff-4f2a-42ef-a11a-d5243a11c5da\") " pod="openshift-must-gather-9px9d/must-gather-s9knr" Nov 03 23:16:53.733778 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.733758 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0920caff-4f2a-42ef-a11a-d5243a11c5da-must-gather-output\") pod \"must-gather-s9knr\" (UID: \"0920caff-4f2a-42ef-a11a-d5243a11c5da\") " pod="openshift-must-gather-9px9d/must-gather-s9knr" Nov 03 23:16:53.789162 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.789091 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w54dp\" (UniqueName: \"kubernetes.io/projected/0920caff-4f2a-42ef-a11a-d5243a11c5da-kube-api-access-w54dp\") pod \"must-gather-s9knr\" (UID: \"0920caff-4f2a-42ef-a11a-d5243a11c5da\") " pod="openshift-must-gather-9px9d/must-gather-s9knr" Nov 03 23:16:53.844407 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:53.844379 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9px9d/must-gather-s9knr" Nov 03 23:16:54.215265 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:54.215229 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9px9d/must-gather-s9knr"] Nov 03 23:16:54.221939 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:16:54.221910 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0920caff_4f2a_42ef_a11a_d5243a11c5da.slice/crio-a1ea9b1fe010443061f09ed48ec716b28a8caf9a461974f66319a18d3b6138fa WatchSource:0}: Error finding container a1ea9b1fe010443061f09ed48ec716b28a8caf9a461974f66319a18d3b6138fa: Status 404 returned error can't find the container with id a1ea9b1fe010443061f09ed48ec716b28a8caf9a461974f66319a18d3b6138fa Nov 03 23:16:54.223793 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:54.223778 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:16:54.917838 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:54.917807 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9px9d/must-gather-s9knr" event={"ID":"0920caff-4f2a-42ef-a11a-d5243a11c5da","Type":"ContainerStarted","Data":"a1ea9b1fe010443061f09ed48ec716b28a8caf9a461974f66319a18d3b6138fa"} Nov 03 23:16:55.923785 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:55.923746 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9px9d/must-gather-s9knr" event={"ID":"0920caff-4f2a-42ef-a11a-d5243a11c5da","Type":"ContainerStarted","Data":"bc2d706563ccae864b30ec87d0933b6c81084c637ba528f9ff17a28f1edaf859"} Nov 03 23:16:55.924125 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:55.923792 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9px9d/must-gather-s9knr" event={"ID":"0920caff-4f2a-42ef-a11a-d5243a11c5da","Type":"ContainerStarted","Data":"11ee39e64b8fdf7001f677e95e41f31b2f43db1352db850b42f9360043b1f5ec"} Nov 03 23:16:56.029736 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:16:56.029610 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9px9d/must-gather-s9knr" podStartSLOduration=1.568426433 podStartE2EDuration="3.029594933s" totalImagesPullingTime="1.4611685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:16:56.029311745 +0000 UTC m=+4135.441497763" watchObservedRunningTime="2026-11-03 23:16:56.029594933 +0000 UTC m=+4135.441780950" Nov 03 23:16:58.995574 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:16:58.995532 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:17:04.029243 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:04.029200 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lwk4s_5b28053b-cac3-4543-a09b-25915e6b3f0c/global-pull-secret-syncer/0.log" Nov 03 23:17:07.120023 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:07.119984 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x78ps_23923c80-00d0-4e1b-adbc-89cc9a992709/konnectivity-agent/0.log" Nov 03 23:17:07.463749 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:07.463711 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-57.us-west-2.compute.internal_56a5599a8af5a3ab98eedd947c314121/haproxy/0.log" Nov 03 23:17:12.708164 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:12.708074 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-656fb666db-8fhmz_ef51a905-8e84-496a-8b7e-02cbdd3a2395/authorino/0.log" Nov 03 23:17:13.512915 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.512879 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w"] Nov 03 23:17:13.521677 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.521642 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.567881 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.567843 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w"] Nov 03 23:17:13.638624 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.638590 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-podres\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.638843 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.638633 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsxh4\" (UniqueName: \"kubernetes.io/projected/9cc66d83-446d-4260-8481-4127c0e8852d-kube-api-access-nsxh4\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.641754 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.639107 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-sys\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.641754 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.639165 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-proc\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.641754 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.639266 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-lib-modules\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.740703 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.740666 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-podres\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.741103 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.740712 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsxh4\" (UniqueName: \"kubernetes.io/projected/9cc66d83-446d-4260-8481-4127c0e8852d-kube-api-access-nsxh4\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.741103 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.740733 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-sys\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.741103 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.740749 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-proc\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.741103 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.740796 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-lib-modules\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.741103 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.740814 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-sys\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.741103 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.740815 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-podres\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.741103 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.740882 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-proc\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.741103 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.740888 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cc66d83-446d-4260-8481-4127c0e8852d-lib-modules\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.795313 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.795215 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsxh4\" (UniqueName: \"kubernetes.io/projected/9cc66d83-446d-4260-8481-4127c0e8852d-kube-api-access-nsxh4\") pod \"perf-node-gather-daemonset-g4t8w\" (UID: \"9cc66d83-446d-4260-8481-4127c0e8852d\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:13.837724 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:13.837697 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:14.212945 ip-10-0-134-57 kubenswrapper[6234]: W1103 23:17:14.212906 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9cc66d83_446d_4260_8481_4127c0e8852d.slice/crio-43644ae623aa44aca800b04dd410395cd41e20d78b791b79c6189b14aa103207 WatchSource:0}: Error finding container 43644ae623aa44aca800b04dd410395cd41e20d78b791b79c6189b14aa103207: Status 404 returned error can't find the container with id 43644ae623aa44aca800b04dd410395cd41e20d78b791b79c6189b14aa103207 Nov 03 23:17:14.247270 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:14.247241 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w"] Nov 03 23:17:14.704251 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:14.704225 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-274zz_f669b176-901b-4865-8735-b04ac3c7dff6/limitador/0.log" Nov 03 23:17:15.064410 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:15.064325 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" event={"ID":"9cc66d83-446d-4260-8481-4127c0e8852d","Type":"ContainerStarted","Data":"22bab4b96d39a9a10778f96042c695018590d95dd18baba3e87fab1c9beb472c"} Nov 03 23:17:15.064410 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:15.064379 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" event={"ID":"9cc66d83-446d-4260-8481-4127c0e8852d","Type":"ContainerStarted","Data":"43644ae623aa44aca800b04dd410395cd41e20d78b791b79c6189b14aa103207"} Nov 03 23:17:15.064836 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:15.064432 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:15.171647 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:15.171591 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" podStartSLOduration=2.171571562 podStartE2EDuration="2.171571562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:15.170981672 +0000 UTC m=+4154.583167690" watchObservedRunningTime="2026-11-03 23:17:15.171571562 +0000 UTC m=+4154.583757570" Nov 03 23:17:21.081685 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:21.081654 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-g4t8w" Nov 03 23:17:21.388596 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:21.388479 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vwb8n_433b8aa2-23d3-432b-abf6-15b7b334b287/kube-state-metrics/0.log" Nov 03 23:17:21.797960 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:21.797921 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vwb8n_433b8aa2-23d3-432b-abf6-15b7b334b287/kube-rbac-proxy-main/0.log" Nov 03 23:17:22.214946 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:22.214914 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vwb8n_433b8aa2-23d3-432b-abf6-15b7b334b287/kube-rbac-proxy-self/0.log" Nov 03 23:17:22.635463 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:22.635377 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-779ff6cb48-8rzf9_5692356c-6609-46d4-9cd7-513a0d883779/metrics-server/0.log" Nov 03 23:17:24.774239 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:24.774207 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wbfjj_02608c76-a776-4f06-94d3-eb3527a9dcdf/node-exporter/0.log" Nov 03 23:17:25.188410 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:25.188379 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wbfjj_02608c76-a776-4f06-94d3-eb3527a9dcdf/kube-rbac-proxy/0.log" Nov 03 23:17:25.605850 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:25.605775 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wbfjj_02608c76-a776-4f06-94d3-eb3527a9dcdf/init-textfile/0.log" Nov 03 23:17:27.264160 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:27.264118 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qf2db_ed75a83a-9f07-4a2f-be7f-1a0b249728c0/kube-rbac-proxy-main/0.log" Nov 03 23:17:27.680329 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:27.680250 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qf2db_ed75a83a-9f07-4a2f-be7f-1a0b249728c0/kube-rbac-proxy-self/0.log" Nov 03 23:17:28.095519 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:28.095463 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qf2db_ed75a83a-9f07-4a2f-be7f-1a0b249728c0/openshift-state-metrics/0.log" Nov 03 23:17:28.996782 ip-10-0-134-57 kubenswrapper[6234]: E1103 23:17:28.996738 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pjmrm" containerName="extractor" Nov 03 23:17:31.612059 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:31.612026 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wkmds_a0f26ebc-df35-404e-9466-9f5d73478ef7/prometheus-operator/0.log" Nov 03 23:17:32.021428 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:32.021397 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wkmds_a0f26ebc-df35-404e-9466-9f5d73478ef7/kube-rbac-proxy/0.log" Nov 03 23:17:32.445089 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:32.445056 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-7zgbj_c9d22708-0a5b-41de-bd34-78e11212e865/prometheus-operator-admission-webhook/0.log" Nov 03 23:17:32.875937 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:32.875845 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bfb5bb58-vkr9f_5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/telemeter-client/0.log" Nov 03 23:17:33.292385 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:33.292352 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bfb5bb58-vkr9f_5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/reload/0.log" Nov 03 23:17:33.707282 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:33.707250 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bfb5bb58-vkr9f_5d71ea07-fa84-4b2a-b06d-f7b05d7a126d/kube-rbac-proxy/0.log" Nov 03 23:17:34.128535 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:34.128449 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddd84555-gsxvd_d962bd32-3fc5-4a46-ba44-0d08701a6cd0/thanos-query/0.log" Nov 03 23:17:34.543830 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:34.543799 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddd84555-gsxvd_d962bd32-3fc5-4a46-ba44-0d08701a6cd0/kube-rbac-proxy-web/0.log" Nov 03 23:17:34.960475 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:34.960452 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddd84555-gsxvd_d962bd32-3fc5-4a46-ba44-0d08701a6cd0/kube-rbac-proxy/0.log" Nov 03 23:17:35.377586 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:35.377484 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddd84555-gsxvd_d962bd32-3fc5-4a46-ba44-0d08701a6cd0/prom-label-proxy/0.log" Nov 03 23:17:35.793202 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:35.793167 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddd84555-gsxvd_d962bd32-3fc5-4a46-ba44-0d08701a6cd0/kube-rbac-proxy-rules/0.log" Nov 03 23:17:36.209087 ip-10-0-134-57 kubenswrapper[6234]: I1103 23:17:36.209062 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddd84555-gsxvd_d962bd32-3fc5-4a46-ba44-0d08701a6cd0/kube-rbac-proxy-metrics/0.log"