Nov 03 05:52:36.090890 ip-10-0-141-67 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 05:52:36.090900 ip-10-0-141-67 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 05:52:36.090907 ip-10-0-141-67 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 05:52:36.091134 ip-10-0-141-67 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 05:52:46.331794 ip-10-0-141-67 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 05:52:46.331809 ip-10-0-141-67 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 45f85e64bb2b460cbecd652a8ffa59c1 -- Nov 03 05:54:30.931645 ip-10-0-141-67 systemd[1]: Starting Kubernetes Kubelet... Nov 03 05:54:31.335293 ip-10-0-141-67 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 05:54:31.335293 ip-10-0-141-67 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 05:54:31.335293 ip-10-0-141-67 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 05:54:31.335293 ip-10-0-141-67 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 05:54:31.340619 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340546 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:54:31.340619 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340619 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:54:31.340619 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340622 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340625 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340629 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340631 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340634 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340638 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340641 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340644 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340646 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340649 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340654 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340658 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340662 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340664 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340667 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340669 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340672 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340674 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340678 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:54:31.340712 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340681 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340684 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340686 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340689 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340692 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340694 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340697 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340700 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340703 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340705 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340708 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340710 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340713 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340716 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340718 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340721 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340724 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340727 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340729 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340732 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:54:31.341143 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340735 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340737 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340740 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340743 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340745 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340748 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340750 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340753 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340755 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340758 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340760 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340763 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340765 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340768 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340771 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340774 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340776 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340779 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340781 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340784 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:54:31.341643 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340786 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340790 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340792 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340795 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340797 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340800 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340802 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340805 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340807 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340810 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340812 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340815 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340817 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340820 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340824 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340829 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340833 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340836 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340839 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:54:31.342106 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340843 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340845 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340848 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340851 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340853 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340856 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340859 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340862 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340864 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340867 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340870 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340872 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340875 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340877 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340880 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340883 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340886 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340888 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340891 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:54:31.342602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340894 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340896 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340899 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340901 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340905 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340907 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.340911 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341751 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341758 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341762 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341770 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341773 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341776 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341779 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341781 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341784 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341786 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341789 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341791 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341795 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:54:31.343044 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341798 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341801 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341804 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341806 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341809 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341812 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341814 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341817 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341820 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341822 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341825 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341828 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341830 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341832 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341835 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341837 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341842 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341845 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341848 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:54:31.343633 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341851 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341854 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341856 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341859 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341867 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341870 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341872 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341875 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341878 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341880 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341883 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341885 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341888 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341891 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341894 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341896 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341899 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341902 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341904 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:54:31.344084 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341906 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341909 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341912 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341914 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341917 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341919 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341922 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341924 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341926 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341929 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341931 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341934 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341936 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341939 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341941 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341944 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341946 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341949 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341956 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341959 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:54:31.344544 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341967 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341970 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341972 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341975 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341977 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341981 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341984 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341986 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341989 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341991 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341994 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341996 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.341999 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342001 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342004 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342006 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342009 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342012 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342014 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:54:31.345013 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342017 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342020 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342022 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342025 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342027 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342029 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342032 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342035 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342037 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342039 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342042 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342046 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342055 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342059 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342062 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342064 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342641 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342649 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342652 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:54:31.345494 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342656 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342659 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342662 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342666 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342669 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342672 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342675 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342677 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342680 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342683 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342686 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342688 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342691 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342694 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342696 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342699 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342702 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342704 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342707 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342710 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:54:31.345942 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342712 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342714 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342717 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342719 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342722 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342725 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342727 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342730 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342732 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342735 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342737 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342740 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342742 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342745 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342748 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342750 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342753 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342756 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342758 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:54:31.346442 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342761 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342764 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342766 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342769 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342771 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342774 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342777 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342780 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342785 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342788 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342790 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342793 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342795 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342798 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342801 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342803 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342806 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342808 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342811 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342814 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:54:31.346897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342817 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342819 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342822 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342824 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342827 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342829 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342834 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342837 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342839 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342842 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342845 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342847 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342850 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342852 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342855 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342858 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342860 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342863 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342865 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:54:31.347360 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342868 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342871 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342874 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342878 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342881 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342884 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342887 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342890 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342893 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342895 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342898 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342900 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342904 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342907 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342910 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342912 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342914 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342917 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342920 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:54:31.347852 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342925 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342927 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342930 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342932 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342935 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.342938 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343031 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343041 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343050 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343057 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343064 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343068 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343073 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343077 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343082 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343085 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343088 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343092 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343095 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343098 2581 flags.go:64] FLAG: --cgroup-root="" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343101 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343104 2581 flags.go:64] FLAG: --client-ca-file="" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343107 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 03 05:54:31.348293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343110 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343116 2581 flags.go:64] FLAG: --cluster-domain="" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343119 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343122 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343126 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343129 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343134 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343137 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343140 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343144 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343147 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343150 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343153 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343156 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343159 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343163 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343166 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343169 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343172 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343175 2581 flags.go:64] FLAG: --enable-server="true" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343178 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343182 2581 flags.go:64] FLAG: --event-burst="100" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343185 2581 flags.go:64] FLAG: --event-qps="50" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343189 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 05:54:31.348835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343193 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343195 2581 flags.go:64] FLAG: --eviction-hard="" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343200 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343203 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343205 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343209 2581 flags.go:64] FLAG: --eviction-soft="" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343212 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343215 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343218 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343221 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343224 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343227 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343230 2581 flags.go:64] FLAG: --feature-gates="" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343234 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343238 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343241 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343244 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343248 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343252 2581 flags.go:64] FLAG: --help="false" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343255 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343258 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343261 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343264 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343268 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 05:54:31.349479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343271 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343274 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343277 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343280 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343283 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343286 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343289 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343294 2581 flags.go:64] FLAG: --kube-reserved="" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343297 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343300 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343303 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343306 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343309 2581 flags.go:64] FLAG: --lock-file="" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343312 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343315 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343318 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343324 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343326 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343330 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343333 2581 flags.go:64] FLAG: --logging-format="text" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343336 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343339 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343342 2581 flags.go:64] FLAG: --manifest-url="" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343345 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343350 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 05:54:31.350092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343354 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343358 2581 flags.go:64] FLAG: --max-pods="110" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343362 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343365 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343368 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343371 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343374 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343377 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343380 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343405 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343409 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343412 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343415 2581 flags.go:64] FLAG: --pod-cidr="" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343418 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343423 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343426 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343429 2581 flags.go:64] FLAG: --port="10250" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343432 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343435 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09a764ebb1f190fa9" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343438 2581 flags.go:64] FLAG: --qos-reserved="" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343441 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343444 2581 flags.go:64] FLAG: --register-node="true" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343447 2581 flags.go:64] FLAG: --register-with-taints="" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343451 2581 flags.go:64] FLAG: --registry-burst="10" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343455 2581 flags.go:64] FLAG: --registry-qps="5" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343457 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 03 05:54:31.351175 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343460 2581 flags.go:64] FLAG: --reserved-memory="" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343464 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343467 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343471 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343474 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343477 2581 flags.go:64] FLAG: --runonce="false" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343482 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343485 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343488 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343491 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343494 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343497 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343500 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343503 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343506 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343509 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343512 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343514 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343517 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343520 2581 flags.go:64] FLAG: --system-cgroups="" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343525 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343530 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343533 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343536 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343541 2581 flags.go:64] FLAG: --tls-min-version="" Nov 03 05:54:31.352062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343544 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343547 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343549 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343552 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343555 2581 flags.go:64] FLAG: --v="2" Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343560 2581 flags.go:64] FLAG: --version="false" Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343564 2581 flags.go:64] FLAG: --vmodule="" Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343568 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.343572 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343708 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343712 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343716 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343720 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343725 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343728 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343731 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343733 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343736 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343739 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343742 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343744 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343747 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:54:31.352662 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343749 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343752 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343754 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343757 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343760 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343763 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343766 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343769 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343772 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343774 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343777 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343780 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343782 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343785 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343788 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343790 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343793 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343795 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343798 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:54:31.353189 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343800 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343803 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343806 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343808 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343812 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343815 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343818 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343820 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343823 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343826 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343828 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343831 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343833 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343836 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343839 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343841 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343844 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343847 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343851 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343854 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:54:31.353688 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343856 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343859 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343861 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343864 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343866 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343869 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343872 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343874 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343876 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343879 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343883 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343886 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343890 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343893 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343896 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343899 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343902 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343908 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343911 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:54:31.354163 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343914 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343916 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343919 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343921 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343924 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343926 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343929 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343931 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343934 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343936 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343939 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343943 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343945 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343948 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343950 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343953 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343956 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343958 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343961 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343963 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:54:31.354653 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343966 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343969 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343971 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343974 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343977 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343981 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343984 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343987 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343990 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343994 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.343996 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344000 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344003 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344005 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344008 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.344034 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 05:54:31.355132 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344072 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344076 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344080 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344082 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344085 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344088 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344090 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344095 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344097 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344100 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344102 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344105 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344108 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344111 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344113 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344116 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344118 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344121 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344123 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:54:31.355545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344126 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344129 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344131 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344134 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344136 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344139 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344142 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344145 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344148 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344151 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344155 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344159 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344162 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344165 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344167 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344170 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344172 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344175 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344177 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344180 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:54:31.355992 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344184 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344186 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344189 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344191 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344194 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344196 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344199 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344201 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344204 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344206 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344209 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344211 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344214 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344216 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344219 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344222 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344224 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344227 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344229 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:54:31.356470 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344232 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344235 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344238 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344241 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344243 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344246 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344248 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344251 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344253 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344256 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344258 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344261 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344263 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344268 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344270 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344273 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344276 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344278 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344281 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:54:31.356925 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344283 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344286 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344288 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344291 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344293 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344296 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344298 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344301 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344304 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344306 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344309 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344311 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344314 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344317 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344319 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344322 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344324 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344327 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344330 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344332 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:54:31.357467 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344335 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344337 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344340 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344343 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344347 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344351 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344355 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344358 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.344360 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.353282 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.353300 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.353320 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.353325 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353405 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353412 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353416 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353419 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353423 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353426 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:54:31.357934 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353429 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353432 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353434 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353438 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353440 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353443 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353446 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353448 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353451 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353454 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353457 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353460 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353462 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353465 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353468 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353471 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353474 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353477 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353480 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:54:31.358423 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353484 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353488 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353490 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353494 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353497 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353500 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353503 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353506 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353508 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353511 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353514 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353517 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353519 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353522 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353525 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353527 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353530 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353533 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353535 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353538 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:54:31.358899 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353541 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353544 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353547 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353549 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353552 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353555 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353557 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353560 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353562 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353565 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353568 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353570 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353573 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353575 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353578 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353581 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353584 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353586 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353589 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353592 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:54:31.359369 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353594 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353597 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353600 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353602 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353605 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353608 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353610 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353613 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353616 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353618 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353621 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353623 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353626 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353628 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353632 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353634 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353637 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353640 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353642 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:54:31.359876 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353645 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353648 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353650 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353653 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353656 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353658 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353661 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353664 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353667 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353670 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353674 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353678 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353682 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353684 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353688 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353691 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353693 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353696 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353700 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:54:31.360342 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353703 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353705 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353708 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.353738 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353887 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353894 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353898 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353901 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353904 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353908 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353911 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353913 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353916 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353919 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353921 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353924 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:54:31.360816 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353927 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353930 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353932 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353935 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353937 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353940 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353944 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353946 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353949 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353952 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353954 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353957 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353959 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353962 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353965 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353968 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353970 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353972 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353975 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353978 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:54:31.361203 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353980 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353982 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353985 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353988 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353991 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353993 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.353997 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354000 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354002 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354005 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354007 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354010 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354013 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354016 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354018 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354021 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354023 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354026 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354030 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:54:31.361713 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354035 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354038 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354041 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354043 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354046 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354048 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354051 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354054 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354056 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354059 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354061 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354064 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354066 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354069 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354072 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354074 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354077 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354079 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354082 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:54:31.362187 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354084 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354087 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354090 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354093 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354095 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354098 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354101 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354103 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354106 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354109 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354111 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354114 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354117 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354119 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354122 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354124 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354127 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354129 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354132 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354135 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:54:31.362725 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354137 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354140 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354143 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354145 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354148 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354150 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354154 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354158 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354161 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354164 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354166 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354169 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354171 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354174 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354178 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:54:31.363214 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354180 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.354204 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354235 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354239 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354242 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354245 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354248 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354251 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354254 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354256 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354259 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354262 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354264 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354267 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354269 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354272 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:54:31.363602 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354275 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354278 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354281 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354284 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354287 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354289 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354292 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354294 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354297 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354299 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354303 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354307 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354310 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354313 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354316 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354318 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354322 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354324 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354327 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:54:31.363973 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354329 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354332 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354335 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354337 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354340 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354343 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354345 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354348 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354350 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354353 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354355 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354358 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354361 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354363 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354366 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354369 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354371 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354374 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354376 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354380 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:54:31.364426 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354382 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354398 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354402 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354406 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354409 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354413 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354417 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354421 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354425 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354428 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354431 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354434 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354437 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354439 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354442 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354445 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354448 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354451 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354453 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354456 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:54:31.364937 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354459 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354464 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354467 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354469 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354472 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354475 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354477 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354480 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354483 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354485 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354488 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354491 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354496 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354498 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354501 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354504 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354506 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354509 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354512 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:54:31.365454 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354514 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354517 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354519 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354522 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354525 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354528 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354530 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354533 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354537 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354539 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354542 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354544 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354547 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:31.354549 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.354609 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.357088 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.358175 2581 server.go:1014] "Starting client certificate rotation" Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.358272 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 05:54:31.365897 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.358306 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 05:54:31.366319 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.360433 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 05:54:31.366319 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.361577 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 05:54:31.373721 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.373703 2581 log.go:25] "Validated CRI v1 runtime API" Nov 03 05:54:31.377734 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.377718 2581 log.go:25] "Validated CRI v1 image API" Nov 03 05:54:31.378581 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.378551 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 05:54:31.381397 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.381362 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9c6b24d3-750a-44d5-bd68-7a6d348f221c:/dev/nvme0n1p3 c68664e9-c913-479b-b5cc-616a813a84e1:/dev/nvme0n1p4] Nov 03 05:54:31.381461 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.381401 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 05:54:31.388651 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.388534 2581 manager.go:217] Machine: {Timestamp:2026-11-03 05:54:31.386990801 +0000 UTC m=+0.367140416 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099466 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec253fb47201fedfc553bc4d6b160c53 SystemUUID:ec253fb4-7201-fedf-c553-bc4d6b160c53 BootID:45f85e64-bb2b-460c-becd-652a8ffa59c1 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:fc:0f:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:fc:0f:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:08:38:50:35:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 05:54:31.388651 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.388642 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 05:54:31.388789 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.388721 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 05:54:31.392443 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.392422 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 05:54:31.392580 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.392445 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-67.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 05:54:31.392621 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.392582 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 05:54:31.392621 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.392590 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 05:54:31.392621 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.392615 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 05:54:31.392698 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.392631 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 05:54:31.393928 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.393912 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 05:54:31.398673 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.398660 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 05:54:31.398798 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.398788 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 05:54:31.399603 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.399593 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 03 05:54:31.399664 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.399606 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 05:54:31.399664 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.399620 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 05:54:31.399664 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.399629 2581 kubelet.go:405] "Adding apiserver pod source" Nov 03 05:54:31.399664 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.399638 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 05:54:31.399960 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.399945 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 05:54:31.403693 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.403673 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 05:54:31.411272 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.411259 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 05:54:31.412216 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.412204 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 05:54:31.412254 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.412235 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 05:54:31.412469 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.412456 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 05:54:31.412499 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.412478 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 05:54:31.412499 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.412487 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 05:54:31.412499 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.412495 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 05:54:31.412589 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.412504 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 05:54:31.412589 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.412513 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 05:54:31.413119 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.413109 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 05:54:31.413147 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.413120 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 05:54:31.413147 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.413128 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 05:54:31.413147 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.413134 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 05:54:31.413147 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.413143 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 05:54:31.413301 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.413152 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 05:54:31.414827 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.414817 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 05:54:31.414859 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.414831 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 05:54:31.418977 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.418965 2581 server.go:1263] "Started kubelet" Nov 03 05:54:31.419080 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.419043 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 05:54:31.419134 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.419058 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 05:54:31.419189 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.419142 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 05:54:31.419886 ip-10-0-141-67 systemd[1]: Started Kubernetes Kubelet. Nov 03 05:54:31.420424 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.420409 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 05:54:31.424140 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.423995 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 03 05:54:31.427179 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.427162 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 05:54:31.431514 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.431489 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 05:54:31.432232 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.432218 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 05:54:31.432364 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.432341 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:31.433277 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.433142 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 05:54:31.433492 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.433153 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 05:54:31.433492 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.433493 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 05:54:31.434322 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.433675 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 03 05:54:31.434322 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.433683 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 03 05:54:31.434322 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.434034 2581 factory.go:153] Registering CRI-O factory Nov 03 05:54:31.434322 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.434048 2581 factory.go:223] Registration of the crio container factory successfully Nov 03 05:54:31.434322 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.434104 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 05:54:31.434322 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.434127 2581 factory.go:55] Registering systemd factory Nov 03 05:54:31.434322 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.434136 2581 factory.go:223] Registration of the systemd container factory successfully Nov 03 05:54:31.434322 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.434152 2581 factory.go:103] Registering Raw factory Nov 03 05:54:31.434322 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.434195 2581 manager.go:1199] Started watching for new ooms in manager Nov 03 05:54:31.436004 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.435478 2581 manager.go:319] Starting recovery of all containers Nov 03 05:54:31.437832 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.437808 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 05:54:31.443872 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.443846 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r2zrj" Nov 03 05:54:31.447058 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.447040 2581 manager.go:324] Recovery completed Nov 03 05:54:31.451749 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.451735 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 05:54:31.453603 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.453586 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientMemory" Nov 03 05:54:31.453680 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.453619 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 05:54:31.453680 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.453633 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientPID" Nov 03 05:54:31.453792 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.453764 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-67.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 05:54:31.453907 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.453871 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 05:54:31.454144 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.454129 2581 cpu_manager.go:226] "Starting" policy="none" Nov 03 05:54:31.454144 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.454141 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 05:54:31.454249 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.454142 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 05:54:31.454249 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.454216 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 05:54:31.454892 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.453746 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-67.ec2.internal.18e4734041c6a331 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-67.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-67.ec2.internal,},FirstTimestamp:2026-11-03 05:54:31.418938161 +0000 UTC m=+0.399087777,LastTimestamp:2026-11-03 05:54:31.418938161 +0000 UTC m=+0.399087777,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-67.ec2.internal,}" Nov 03 05:54:31.454985 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.454925 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientMemory" Nov 03 05:54:31.454985 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.454951 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 05:54:31.454985 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.454978 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientPID" Nov 03 05:54:31.456355 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.456343 2581 policy_none.go:50] "Start" Nov 03 05:54:31.456436 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.456359 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 05:54:31.456436 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.456376 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 05:54:31.457837 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.457824 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r2zrj" Nov 03 05:54:31.459097 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.459084 2581 policy_none.go:44] "Start" Nov 03 05:54:31.468271 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.468248 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 05:54:31.468354 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.468280 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 05:54:31.468354 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.468326 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 05:54:31.468475 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.468401 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-67.ec2.internal.18e4734043d79328 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-67.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-67.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-67.ec2.internal,},FirstTimestamp:2026-11-03 05:54:31.4536026 +0000 UTC m=+0.433752220,LastTimestamp:2026-11-03 05:54:31.4536026 +0000 UTC m=+0.433752220,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-67.ec2.internal,}" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.498952 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.501953 2581 manager.go:341] "Starting Device Plugin manager" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.501986 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.501997 2581 server.go:87] "Starting device plugin registration server" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.502271 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.502514 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.502612 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.502704 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.502712 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.503190 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.503230 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.510332 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.511518 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.512770 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.512801 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 05:54:31.514685 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.512838 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 05:54:31.528737 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.528718 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 05:54:31.603244 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.603195 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 05:54:31.604105 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.604084 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientMemory" Nov 03 05:54:31.604192 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.604126 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 05:54:31.604192 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.604142 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientPID" Nov 03 05:54:31.604192 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.604172 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.613351 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.613321 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal"] Nov 03 05:54:31.613416 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.613405 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 05:54:31.614196 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.614179 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientMemory" Nov 03 05:54:31.614259 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.614206 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 05:54:31.614259 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.614215 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientPID" Nov 03 05:54:31.616440 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.616428 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 05:54:31.616632 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.616584 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.616632 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.616623 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 05:54:31.617132 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.617117 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientMemory" Nov 03 05:54:31.617200 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.617144 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 05:54:31.617200 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.617121 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientMemory" Nov 03 05:54:31.617200 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.617154 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientPID" Nov 03 05:54:31.617200 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.617171 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 05:54:31.617200 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.617182 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientPID" Nov 03 05:54:31.619248 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.619229 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.619330 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.619259 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 05:54:31.620642 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.620629 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientMemory" Nov 03 05:54:31.620718 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.620650 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 05:54:31.620718 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.620660 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientPID" Nov 03 05:54:31.624702 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.624685 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.624791 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.624710 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-67.ec2.internal\": node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:31.635047 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.635019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.635152 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.635060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.635152 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.635084 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45ef46b1ad95006153c299e1a6c8cea2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-67.ec2.internal\" (UID: \"45ef46b1ad95006153c299e1a6c8cea2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.653049 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.653028 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-67.ec2.internal\" not found" node="ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.655668 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.655648 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-67.ec2.internal\" not found" node="ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.670641 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.670623 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:31.736210 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.736188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45ef46b1ad95006153c299e1a6c8cea2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-67.ec2.internal\" (UID: \"45ef46b1ad95006153c299e1a6c8cea2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.736311 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.736212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.736311 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.736236 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.736311 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.736285 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.736311 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.736298 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.736479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.736298 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45ef46b1ad95006153c299e1a6c8cea2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-67.ec2.internal\" (UID: \"45ef46b1ad95006153c299e1a6c8cea2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.771537 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.771516 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:31.872589 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.872518 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:31.958293 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.958264 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.960479 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:31.960459 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:54:31.972884 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:31.972866 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:32.073420 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:32.073380 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:32.174757 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:32.174722 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:32.275753 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:32.275718 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:32.358214 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.358188 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 05:54:32.358714 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.358387 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 05:54:32.376626 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:32.376597 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:32.463515 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.463476 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 05:49:32 +0000 UTC" deadline="2028-06-05 00:16:48.947882829 +0000 UTC" Nov 03 05:54:32.463515 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.463510 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13914h22m16.484376256s" Nov 03 05:54:32.477383 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:32.477363 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:32.547350 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.547330 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 05:54:32.548634 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.548611 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientMemory" Nov 03 05:54:32.548748 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.548650 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 05:54:32.548748 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.548660 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientPID" Nov 03 05:54:32.552566 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:32.552538 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ef46b1ad95006153c299e1a6c8cea2.slice/crio-649c95861450533aae74aa2e6b4a98dd10706551a5f6c510e57802b6561ad0f5 WatchSource:0}: Error finding container 649c95861450533aae74aa2e6b4a98dd10706551a5f6c510e57802b6561ad0f5: Status 404 returned error can't find the container with id 649c95861450533aae74aa2e6b4a98dd10706551a5f6c510e57802b6561ad0f5 Nov 03 05:54:32.553569 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:32.553548 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b012b3f595caea8c478eb61ba981af.slice/crio-ee9b430004788822fe9777f5b33e99c1c93ea7f922c7ddac81d34c7980535505 WatchSource:0}: Error finding container ee9b430004788822fe9777f5b33e99c1c93ea7f922c7ddac81d34c7980535505: Status 404 returned error can't find the container with id ee9b430004788822fe9777f5b33e99c1c93ea7f922c7ddac81d34c7980535505 Nov 03 05:54:32.559668 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.559653 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 05:54:32.578065 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:32.578036 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:32.678319 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:32.678234 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:32.781461 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:32.781429 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:32.883250 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:32.883218 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:54:32.900287 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.900252 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 05:54:32.932533 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.932465 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:54:32.959150 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.959118 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 05:54:32.961539 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.961515 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:54:32.984676 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:32.984652 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 05:54:33.027951 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.027921 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 05:54:33.376682 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.376641 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 05:54:33.400847 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.400820 2581 apiserver.go:52] "Watching apiserver" Nov 03 05:54:33.432193 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.432159 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 05:54:33.433068 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.433048 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 05:54:33.433373 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.433351 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-c75h9","openshift-multus/network-metrics-daemon-hbdpd","openshift-network-diagnostics/network-check-target-cp5xp","kube-system/global-pull-secret-syncer-vspq2","kube-system/konnectivity-agent-sp2qh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg","openshift-cluster-node-tuning-operator/tuned-kkvkc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal","openshift-dns/node-resolver-99n4t","openshift-multus/multus-w4bpx","openshift-network-operator/iptables-alerter-xzjql","openshift-image-registry/node-ca-vvkcc","openshift-multus/multus-additional-cni-plugins-9c7rn"] Nov 03 05:54:33.436414 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.436388 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.438741 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.438719 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.440912 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.440891 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:33.441024 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.440974 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:33.443238 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.443203 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:33.443345 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.443284 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445514 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysconfig\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445553 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-tuned\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445581 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxwvv\" (UniqueName: \"kubernetes.io/projected/85bfd499-9f29-4d6e-b2de-094d011b4f0b-kube-api-access-jxwvv\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-run\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445613 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445631 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-var-lib-kubelet\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445655 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-tmp\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445679 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-modprobe-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445702 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-kubernetes\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-conf\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445750 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-sys\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445764 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-systemd\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-lib-modules\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.446006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.445878 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-host\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.453018 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.452990 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.453121 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.453076 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.455112 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.455091 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.457108 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.457084 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 05:54:33.457218 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.457198 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 05:54:33.457286 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.457234 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-krwzb\"" Nov 03 05:54:33.458105 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.457370 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z796t\"" Nov 03 05:54:33.458105 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.457485 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:54:33.458105 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.457803 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 05:54:33.458105 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.458075 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 05:54:33.459406 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.459357 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:33.459542 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.459518 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:33.461746 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.461500 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.464572 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.464165 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.467393 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.467371 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.500821 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.500647 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 05:54:33.500821 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.500699 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bgjk9" Nov 03 05:54:33.518005 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.517960 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" event={"ID":"18b012b3f595caea8c478eb61ba981af","Type":"ContainerStarted","Data":"ee9b430004788822fe9777f5b33e99c1c93ea7f922c7ddac81d34c7980535505"} Nov 03 05:54:33.519321 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.519299 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" event={"ID":"45ef46b1ad95006153c299e1a6c8cea2","Type":"ContainerStarted","Data":"649c95861450533aae74aa2e6b4a98dd10706551a5f6c510e57802b6561ad0f5"} Nov 03 05:54:33.534894 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.534874 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 05:54:33.546740 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.546714 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-node-log\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.546867 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.546757 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-lib-modules\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.546867 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.546788 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxwvv\" (UniqueName: \"kubernetes.io/projected/85bfd499-9f29-4d6e-b2de-094d011b4f0b-kube-api-access-jxwvv\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.546962 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.546886 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-bin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.546962 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.546924 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-lib-modules\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.546962 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.546922 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-multus-daemon-config\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.547095 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.546965 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7502a388-bbcd-4916-8b8d-a4258c58d575-agent-certs\") pod \"konnectivity-agent-sp2qh\" (UID: \"7502a388-bbcd-4916-8b8d-a4258c58d575\") " pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:33.547095 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlrf\" (UniqueName: \"kubernetes.io/projected/1eb02066-c20a-45b3-885a-0e9834b1c998-kube-api-access-xhlrf\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.547095 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547052 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-run\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.547095 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547079 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-var-lib-kubelet\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.547249 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547171 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7502a388-bbcd-4916-8b8d-a4258c58d575-konnectivity-ca\") pod \"konnectivity-agent-sp2qh\" (UID: \"7502a388-bbcd-4916-8b8d-a4258c58d575\") " pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:33.547249 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.547249 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547216 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-modprobe-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.547249 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-var-lib-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.547418 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547260 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzl4\" (UniqueName: \"kubernetes.io/projected/2104714e-6924-4133-90b9-7d98135901e8-kube-api-access-5jzl4\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.547418 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547231 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-var-lib-kubelet\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.547418 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-cnibin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.547418 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547291 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-modprobe-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.547418 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547213 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-run\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.547418 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547318 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-hostroot\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.547418 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwqc\" (UniqueName: \"kubernetes.io/projected/5732cb6f-524c-4389-8fe2-f81885322ae5-kube-api-access-2mwqc\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.547742 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1eb02066-c20a-45b3-885a-0e9834b1c998-host\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.547742 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547525 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-conf-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.547742 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-etc-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.547742 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysconfig\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.547742 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-kubelet\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.547742 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547660 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysconfig\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.547742 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547689 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89mmd\" (UniqueName: \"kubernetes.io/projected/0b722d0a-df66-409f-a7cf-0a891f5a1d32-kube-api-access-89mmd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.547742 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547723 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36beafc8-4aee-448b-8dfd-64f89637aa9f-hosts-file\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.547742 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-cni-binary-copy\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-netns\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547808 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2104714e-6924-4133-90b9-7d98135901e8-iptables-alerter-script\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547830 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2104714e-6924-4133-90b9-7d98135901e8-host-slash\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547870 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-systemd-units\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547911 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-socket-dir-parent\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547948 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-k8s-cni-cncf-io\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547972 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqbfm\" (UniqueName: \"kubernetes.io/projected/2f174941-0801-466a-8cef-c2da4faa916f-kube-api-access-cqbfm\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.547996 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-system-cni-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548022 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-systemd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548079 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-ovn\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.548162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-bin\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548178 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-kubelet\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548225 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548255 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-env-overrides\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovn-node-metrics-cert\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548357 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjm42\" (UniqueName: \"kubernetes.io/projected/36beafc8-4aee-448b-8dfd-64f89637aa9f-kube-api-access-cjm42\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548388 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-tmp\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-script-lib\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548493 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67xpb\" (UniqueName: \"kubernetes.io/projected/893a8c11-60f6-4736-9971-10520459a1d0-kube-api-access-67xpb\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548521 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsz5c\" (UniqueName: \"kubernetes.io/projected/58637e90-c0b9-44da-91e9-7e1075a2636e-kube-api-access-gsz5c\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:33.548559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548549 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-conf\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-sys\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548681 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548714 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-log-socket\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548770 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1eb02066-c20a-45b3-885a-0e9834b1c998-serviceca\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548797 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-kubelet-config\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548820 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-dbus\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548882 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-netd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548908 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36beafc8-4aee-448b-8dfd-64f89637aa9f-tmp-dir\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.548985 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-host\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.549037 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-tuned\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549066 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-config\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549236 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-netns\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549296 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549324 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549355 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-multus\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549392 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-multus-certs\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549418 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-etc-kubernetes\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549446 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-cnibin\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549486 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-kubernetes\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549545 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-slash\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549571 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-system-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549582 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-sys\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.549660 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549599 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-device-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.550328 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549628 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.550328 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-systemd\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.550328 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549706 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-binary-copy\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.550328 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549854 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.550328 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549878 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 05:54:33.550328 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.549948 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-kubernetes\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.550328 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.550037 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-systemd\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.550328 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.550164 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-conf\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.550673 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.550458 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-host\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.553892 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.553838 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-tmp\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.554009 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.553897 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-tuned\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.563550 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.563523 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 05:54:33.563645 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.563602 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 05:54:33.563692 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.563676 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 05:54:33.563874 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.563859 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 05:54:33.564266 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564249 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 05:54:33.564353 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564296 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 05:54:33.564424 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564352 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 05:54:33.564508 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564465 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nvbtr\"" Nov 03 05:54:33.564508 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564515 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 05:54:33.564649 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564581 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 05:54:33.564695 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564657 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 05:54:33.564762 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564741 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 05:54:33.564864 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564817 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 05:54:33.564950 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564934 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gfnmx\"" Nov 03 05:54:33.565003 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.564990 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 05:54:33.565085 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.565071 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zq86f\"" Nov 03 05:54:33.565143 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.565133 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 05:54:33.565193 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.565170 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:54:33.565269 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.565255 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-75slc\"" Nov 03 05:54:33.565351 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.565339 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 05:54:33.565413 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.565400 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 05:54:33.566006 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.565945 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 05:54:33.566154 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.566062 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fkxng\"" Nov 03 05:54:33.566154 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.566124 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 05:54:33.566318 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.566187 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 05:54:33.566318 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.566278 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 05:54:33.566459 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.566351 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8hrbq\"" Nov 03 05:54:33.566459 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.566446 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nphqj\"" Nov 03 05:54:33.566567 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.566530 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 05:54:33.574114 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.574088 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxwvv\" (UniqueName: \"kubernetes.io/projected/85bfd499-9f29-4d6e-b2de-094d011b4f0b-kube-api-access-jxwvv\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2104714e-6924-4133-90b9-7d98135901e8-iptables-alerter-script\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2104714e-6924-4133-90b9-7d98135901e8-host-slash\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-systemd-units\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-socket-dir-parent\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-k8s-cni-cncf-io\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqbfm\" (UniqueName: \"kubernetes.io/projected/2f174941-0801-466a-8cef-c2da4faa916f-kube-api-access-cqbfm\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650661 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-system-cni-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-systemd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-ovn\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650717 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-bin\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-kubelet\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650756 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650779 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650796 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-env-overrides\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovn-node-metrics-cert\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650827 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjm42\" (UniqueName: \"kubernetes.io/projected/36beafc8-4aee-448b-8dfd-64f89637aa9f-kube-api-access-cjm42\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.652299 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-script-lib\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650860 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67xpb\" (UniqueName: \"kubernetes.io/projected/893a8c11-60f6-4736-9971-10520459a1d0-kube-api-access-67xpb\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsz5c\" (UniqueName: \"kubernetes.io/projected/58637e90-c0b9-44da-91e9-7e1075a2636e-kube-api-access-gsz5c\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650922 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-ovn\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-log-socket\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.650988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651014 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1eb02066-c20a-45b3-885a-0e9834b1c998-serviceca\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-kubelet-config\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651048 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-dbus\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-netd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651081 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36beafc8-4aee-448b-8dfd-64f89637aa9f-tmp-dir\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651130 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651154 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-config\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.653181 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651192 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651216 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-netns\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651253 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-multus\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651319 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-multus-certs\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651339 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-etc-kubernetes\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-cnibin\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651373 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651390 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-slash\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651410 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-system-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651434 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-device-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651485 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-binary-copy\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651511 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-node-log\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651566 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2104714e-6924-4133-90b9-7d98135901e8-host-slash\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.654057 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651577 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-node-log\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651627 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-bin\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651656 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651664 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-systemd-units\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651709 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-socket-dir-parent\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651741 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-system-cni-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651770 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-systemd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651832 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36beafc8-4aee-448b-8dfd-64f89637aa9f-tmp-dir\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651838 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.651937 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.652125 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.652166 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-log-socket\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.652229 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.653376 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-config\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.653527 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-netd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.653785 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-script-lib\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.654154 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-env-overrides\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.654488 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1eb02066-c20a-45b3-885a-0e9834b1c998-serviceca\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.654834 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.654520 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2104714e-6924-4133-90b9-7d98135901e8-iptables-alerter-script\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.654549 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-k8s-cni-cncf-io\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.654662 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.654744 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.654802 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-kubelet\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.654873 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovn-node-metrics-cert\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.654886 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-kubelet-config\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655022 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-etc-kubernetes\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655075 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-device-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655079 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-dbus\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655104 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-bin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-multus-daemon-config\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655164 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7502a388-bbcd-4916-8b8d-a4258c58d575-agent-certs\") pod \"konnectivity-agent-sp2qh\" (UID: \"7502a388-bbcd-4916-8b8d-a4258c58d575\") " pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655163 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-system-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655205 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-bin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.655276 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655288 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-netns\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.655873 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655339 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-cnibin\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.655358 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret podName:d73ab4e1-b560-420b-8216-6272bd184f39 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:34.155336756 +0000 UTC m=+3.135486384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret") pod "global-pull-secret-syncer-vspq2" (UID: "d73ab4e1-b560-420b-8216-6272bd184f39") : object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655399 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.655936 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-multus-daemon-config\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656006 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656076 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.656105 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656145 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-slash\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.656166 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:54:34.156152218 +0000 UTC m=+3.136301822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656293 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-multus\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656624 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-binary-copy\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656676 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlrf\" (UniqueName: \"kubernetes.io/projected/1eb02066-c20a-45b3-885a-0e9834b1c998-kube-api-access-xhlrf\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.656735 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7502a388-bbcd-4916-8b8d-a4258c58d575-konnectivity-ca\") pod \"konnectivity-agent-sp2qh\" (UID: \"7502a388-bbcd-4916-8b8d-a4258c58d575\") " pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-var-lib-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656821 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzl4\" (UniqueName: \"kubernetes.io/projected/2104714e-6924-4133-90b9-7d98135901e8-kube-api-access-5jzl4\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-cnibin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656866 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-hostroot\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656895 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwqc\" (UniqueName: \"kubernetes.io/projected/5732cb6f-524c-4389-8fe2-f81885322ae5-kube-api-access-2mwqc\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656926 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1eb02066-c20a-45b3-885a-0e9834b1c998-host\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-conf-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656966 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-etc-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.656992 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-kubelet\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.657015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89mmd\" (UniqueName: \"kubernetes.io/projected/0b722d0a-df66-409f-a7cf-0a891f5a1d32-kube-api-access-89mmd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.657047 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36beafc8-4aee-448b-8dfd-64f89637aa9f-hosts-file\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.657079 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-cni-binary-copy\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.657105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-netns\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.657357 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.657188 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-netns\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.658114 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.657910 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.658662 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.658637 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7502a388-bbcd-4916-8b8d-a4258c58d575-konnectivity-ca\") pod \"konnectivity-agent-sp2qh\" (UID: \"7502a388-bbcd-4916-8b8d-a4258c58d575\") " pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:33.658780 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.658713 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-cnibin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.658780 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.658754 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-hostroot\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.658898 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.658808 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.658898 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.658857 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-var-lib-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.659007 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.658900 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-kubelet\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.659007 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.658934 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1eb02066-c20a-45b3-885a-0e9834b1c998-host\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.659007 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.658971 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-conf-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.659007 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.658990 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7502a388-bbcd-4916-8b8d-a4258c58d575-agent-certs\") pod \"konnectivity-agent-sp2qh\" (UID: \"7502a388-bbcd-4916-8b8d-a4258c58d575\") " pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:33.659207 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.659006 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-etc-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.659207 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.659061 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36beafc8-4aee-448b-8dfd-64f89637aa9f-hosts-file\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.659444 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.659319 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-multus-certs\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.660240 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.659542 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-cni-binary-copy\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.681252 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.681228 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67xpb\" (UniqueName: \"kubernetes.io/projected/893a8c11-60f6-4736-9971-10520459a1d0-kube-api-access-67xpb\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.684743 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.684588 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 05:54:33.684743 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.684618 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 05:54:33.684743 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.684636 2581 projected.go:196] Error preparing data for projected volume kube-api-access-cwsq8 for pod openshift-network-diagnostics/network-check-target-cp5xp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:33.684743 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:33.684715 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8 podName:219a457b-c7b8-4a1d-8f14-50a5d56bb9bd nodeName:}" failed. No retries permitted until 2026-11-03 05:54:34.1846951 +0000 UTC m=+3.164844730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cwsq8" (UniqueName: "kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8") pod "network-check-target-cp5xp" (UID: "219a457b-c7b8-4a1d-8f14-50a5d56bb9bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:33.685288 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.685265 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjm42\" (UniqueName: \"kubernetes.io/projected/36beafc8-4aee-448b-8dfd-64f89637aa9f-kube-api-access-cjm42\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.685618 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.685599 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqbfm\" (UniqueName: \"kubernetes.io/projected/2f174941-0801-466a-8cef-c2da4faa916f-kube-api-access-cqbfm\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.687635 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.687590 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlrf\" (UniqueName: \"kubernetes.io/projected/1eb02066-c20a-45b3-885a-0e9834b1c998-kube-api-access-xhlrf\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.688645 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.688617 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzl4\" (UniqueName: \"kubernetes.io/projected/2104714e-6924-4133-90b9-7d98135901e8-kube-api-access-5jzl4\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.690346 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.690319 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwqc\" (UniqueName: \"kubernetes.io/projected/5732cb6f-524c-4389-8fe2-f81885322ae5-kube-api-access-2mwqc\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.690454 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.690345 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsz5c\" (UniqueName: \"kubernetes.io/projected/58637e90-c0b9-44da-91e9-7e1075a2636e-kube-api-access-gsz5c\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:33.692212 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.692193 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89mmd\" (UniqueName: \"kubernetes.io/projected/0b722d0a-df66-409f-a7cf-0a891f5a1d32-kube-api-access-89mmd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.756174 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.756136 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:54:33.762684 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.762654 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:54:33.769834 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:33.769804 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bfd499_9f29_4d6e_b2de_094d011b4f0b.slice/crio-c5b82915284f00fc0070e241567762d66908b992b3eccb0b1c90019962fd92f5 WatchSource:0}: Error finding container c5b82915284f00fc0070e241567762d66908b992b3eccb0b1c90019962fd92f5: Status 404 returned error can't find the container with id c5b82915284f00fc0070e241567762d66908b992b3eccb0b1c90019962fd92f5 Nov 03 05:54:33.775416 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.775390 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:33.776585 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:33.776558 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb02066_c20a_45b3_885a_0e9834b1c998.slice/crio-6dd0b7a3516914b9370f372e6e54c49b972ea78d5a7ccc80fa468db31d7edd9d WatchSource:0}: Error finding container 6dd0b7a3516914b9370f372e6e54c49b972ea78d5a7ccc80fa468db31d7edd9d: Status 404 returned error can't find the container with id 6dd0b7a3516914b9370f372e6e54c49b972ea78d5a7ccc80fa468db31d7edd9d Nov 03 05:54:33.784195 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.784171 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:54:33.787147 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:33.787110 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7502a388_bbcd_4916_8b8d_a4258c58d575.slice/crio-5ec17c2466431329949305a0981ed793552f6226ff216836e2d1be2e6e904ce1 WatchSource:0}: Error finding container 5ec17c2466431329949305a0981ed793552f6226ff216836e2d1be2e6e904ce1: Status 404 returned error can't find the container with id 5ec17c2466431329949305a0981ed793552f6226ff216836e2d1be2e6e904ce1 Nov 03 05:54:33.796219 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:33.796175 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893a8c11_60f6_4736_9971_10520459a1d0.slice/crio-5624d691f94f024b3a87e1b0882e0b03e3ee89535ab7e8ac3a47a52ebf962e2e WatchSource:0}: Error finding container 5624d691f94f024b3a87e1b0882e0b03e3ee89535ab7e8ac3a47a52ebf962e2e: Status 404 returned error can't find the container with id 5624d691f94f024b3a87e1b0882e0b03e3ee89535ab7e8ac3a47a52ebf962e2e Nov 03 05:54:33.802560 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.802539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:54:33.814732 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.814708 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-99n4t" Nov 03 05:54:33.817647 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:33.817615 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5732cb6f_524c_4389_8fe2_f81885322ae5.slice/crio-f7e9d06a2a1bd320cf8e4d6451881ffe3de535e8906f94c326b01ca451574213 WatchSource:0}: Error finding container f7e9d06a2a1bd320cf8e4d6451881ffe3de535e8906f94c326b01ca451574213: Status 404 returned error can't find the container with id f7e9d06a2a1bd320cf8e4d6451881ffe3de535e8906f94c326b01ca451574213 Nov 03 05:54:33.821478 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.821448 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w4bpx" Nov 03 05:54:33.823438 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:33.823413 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36beafc8_4aee_448b_8dfd_64f89637aa9f.slice/crio-718c715c7de938642fa7a115a4af073f8ac12f8aba6373708322979065e4f003 WatchSource:0}: Error finding container 718c715c7de938642fa7a115a4af073f8ac12f8aba6373708322979065e4f003: Status 404 returned error can't find the container with id 718c715c7de938642fa7a115a4af073f8ac12f8aba6373708322979065e4f003 Nov 03 05:54:33.827081 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.827058 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:33.829188 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.829169 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:54:33.830122 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:33.830091 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f174941_0801_466a_8cef_c2da4faa916f.slice/crio-b549c16578dd87e9bdf0b6ac99e5583d0b7c04dd077ab74fcde8d5a2de7f045c WatchSource:0}: Error finding container b549c16578dd87e9bdf0b6ac99e5583d0b7c04dd077ab74fcde8d5a2de7f045c: Status 404 returned error can't find the container with id b549c16578dd87e9bdf0b6ac99e5583d0b7c04dd077ab74fcde8d5a2de7f045c Nov 03 05:54:33.863521 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:33.863494 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b722d0a_df66_409f_a7cf_0a891f5a1d32.slice/crio-c5f9a19056772b54b900cbd373eca0b0ce02f0484ad370a914463ec0753e00dd WatchSource:0}: Error finding container c5f9a19056772b54b900cbd373eca0b0ce02f0484ad370a914463ec0753e00dd: Status 404 returned error can't find the container with id c5f9a19056772b54b900cbd373eca0b0ce02f0484ad370a914463ec0753e00dd Nov 03 05:54:33.866835 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:54:33.866807 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2104714e_6924_4133_90b9_7d98135901e8.slice/crio-089f26e348dc1b9a0fa160f005b1cfafce0b63c24e1e9fa4597d30eace4648dc WatchSource:0}: Error finding container 089f26e348dc1b9a0fa160f005b1cfafce0b63c24e1e9fa4597d30eace4648dc: Status 404 returned error can't find the container with id 089f26e348dc1b9a0fa160f005b1cfafce0b63c24e1e9fa4597d30eace4648dc Nov 03 05:54:33.928046 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:33.927976 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 05:54:34.164203 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.159808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:34.164203 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.159898 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:34.164203 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:34.160042 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:34.164203 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:34.160113 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret podName:d73ab4e1-b560-420b-8216-6272bd184f39 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:35.160094865 +0000 UTC m=+4.140244471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret") pod "global-pull-secret-syncer-vspq2" (UID: "d73ab4e1-b560-420b-8216-6272bd184f39") : object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:34.164203 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:34.160311 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:34.164203 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:34.160360 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:54:35.160346505 +0000 UTC m=+4.140496124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:34.261751 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.261194 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:34.261751 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:34.261597 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 05:54:34.261751 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:34.261622 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 05:54:34.261751 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:34.261635 2581 projected.go:196] Error preparing data for projected volume kube-api-access-cwsq8 for pod openshift-network-diagnostics/network-check-target-cp5xp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:34.261751 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:34.261691 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8 podName:219a457b-c7b8-4a1d-8f14-50a5d56bb9bd nodeName:}" failed. No retries permitted until 2026-11-03 05:54:35.261672726 +0000 UTC m=+4.241822346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwsq8" (UniqueName: "kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8") pod "network-check-target-cp5xp" (UID: "219a457b-c7b8-4a1d-8f14-50a5d56bb9bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:34.505551 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.505412 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 05:49:34 +0000 UTC" deadline="2028-07-22 10:12:48.442158762 +0000 UTC" Nov 03 05:54:34.505551 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.505483 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15052h18m13.936680361s" Nov 03 05:54:34.526581 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.526544 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" event={"ID":"5732cb6f-524c-4389-8fe2-f81885322ae5","Type":"ContainerStarted","Data":"f7e9d06a2a1bd320cf8e4d6451881ffe3de535e8906f94c326b01ca451574213"} Nov 03 05:54:34.533326 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.533290 2581 generic.go:356] "Generic (PLEG): container finished" podID="18b012b3f595caea8c478eb61ba981af" containerID="7fae73732c0d329a9240c03a2c52dec2e97cc13e547075d7b4c5a95dc31cdf32" exitCode=0 Nov 03 05:54:34.533658 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.533589 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" event={"ID":"18b012b3f595caea8c478eb61ba981af","Type":"ContainerDied","Data":"7fae73732c0d329a9240c03a2c52dec2e97cc13e547075d7b4c5a95dc31cdf32"} Nov 03 05:54:34.540837 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.540759 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzjql" event={"ID":"2104714e-6924-4133-90b9-7d98135901e8","Type":"ContainerStarted","Data":"089f26e348dc1b9a0fa160f005b1cfafce0b63c24e1e9fa4597d30eace4648dc"} Nov 03 05:54:34.543064 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.542954 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" event={"ID":"0b722d0a-df66-409f-a7cf-0a891f5a1d32","Type":"ContainerStarted","Data":"c5f9a19056772b54b900cbd373eca0b0ce02f0484ad370a914463ec0753e00dd"} Nov 03 05:54:34.549752 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.549684 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w4bpx" event={"ID":"2f174941-0801-466a-8cef-c2da4faa916f","Type":"ContainerStarted","Data":"b549c16578dd87e9bdf0b6ac99e5583d0b7c04dd077ab74fcde8d5a2de7f045c"} Nov 03 05:54:34.560105 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.559994 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" event={"ID":"893a8c11-60f6-4736-9971-10520459a1d0","Type":"ContainerStarted","Data":"5624d691f94f024b3a87e1b0882e0b03e3ee89535ab7e8ac3a47a52ebf962e2e"} Nov 03 05:54:34.584512 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.584408 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sp2qh" event={"ID":"7502a388-bbcd-4916-8b8d-a4258c58d575","Type":"ContainerStarted","Data":"5ec17c2466431329949305a0981ed793552f6226ff216836e2d1be2e6e904ce1"} Nov 03 05:54:34.610632 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.610593 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vvkcc" event={"ID":"1eb02066-c20a-45b3-885a-0e9834b1c998","Type":"ContainerStarted","Data":"6dd0b7a3516914b9370f372e6e54c49b972ea78d5a7ccc80fa468db31d7edd9d"} Nov 03 05:54:34.617560 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.617514 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" event={"ID":"85bfd499-9f29-4d6e-b2de-094d011b4f0b","Type":"ContainerStarted","Data":"c5b82915284f00fc0070e241567762d66908b992b3eccb0b1c90019962fd92f5"} Nov 03 05:54:34.640766 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.639986 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" event={"ID":"45ef46b1ad95006153c299e1a6c8cea2","Type":"ContainerStarted","Data":"a7d91914558c7941ee95e7ad87dfe375271878ac1781b9d6d055b1b286965476"} Nov 03 05:54:34.652219 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:34.652152 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-99n4t" event={"ID":"36beafc8-4aee-448b-8dfd-64f89637aa9f","Type":"ContainerStarted","Data":"718c715c7de938642fa7a115a4af073f8ac12f8aba6373708322979065e4f003"} Nov 03 05:54:35.169623 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.169535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:35.169623 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.169623 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:35.169881 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.169766 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:35.169881 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.169832 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:54:37.169811122 +0000 UTC m=+6.149960730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:35.169999 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.169918 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:35.169999 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.169952 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret podName:d73ab4e1-b560-420b-8216-6272bd184f39 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:37.169940413 +0000 UTC m=+6.150090021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret") pod "global-pull-secret-syncer-vspq2" (UID: "d73ab4e1-b560-420b-8216-6272bd184f39") : object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:35.270574 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.270535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:35.270768 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.270740 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 05:54:35.270768 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.270761 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 05:54:35.270888 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.270775 2581 projected.go:196] Error preparing data for projected volume kube-api-access-cwsq8 for pod openshift-network-diagnostics/network-check-target-cp5xp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:35.270888 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.270833 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8 podName:219a457b-c7b8-4a1d-8f14-50a5d56bb9bd nodeName:}" failed. No retries permitted until 2026-11-03 05:54:37.270813882 +0000 UTC m=+6.250963493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwsq8" (UniqueName: "kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8") pod "network-check-target-cp5xp" (UID: "219a457b-c7b8-4a1d-8f14-50a5d56bb9bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:35.505951 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.505854 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 05:49:34 +0000 UTC" deadline="2028-05-30 01:07:46.430640303 +0000 UTC" Nov 03 05:54:35.505951 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.505893 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13771h13m10.92475185s" Nov 03 05:54:35.516500 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.516452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:35.516649 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.516605 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:35.517055 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.517039 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:35.517164 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.517148 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:35.517218 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.517205 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:35.517293 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:35.517276 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:35.668279 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.668199 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" event={"ID":"18b012b3f595caea8c478eb61ba981af","Type":"ContainerStarted","Data":"5a38bc902652725cd42096bba832db2ed4c0ade10e99fd71c9dc38b8e4373bbd"} Nov 03 05:54:35.714125 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.713673 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" podStartSLOduration=2.713653958 podStartE2EDuration="2.713653958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:35.712760943 +0000 UTC m=+4.692910569" watchObservedRunningTime="2026-11-03 05:54:35.713653958 +0000 UTC m=+4.693803589" Nov 03 05:54:35.714125 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:35.713788 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" podStartSLOduration=2.713782157 podStartE2EDuration="2.713782157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:34.684330032 +0000 UTC m=+3.664479666" watchObservedRunningTime="2026-11-03 05:54:35.713782157 +0000 UTC m=+4.693931787" Nov 03 05:54:36.676864 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:36.676796 2581 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="2331d1819ab747057808128e9cb00b4554cac81b2a29a6a548e0f970256ed234" exitCode=0 Nov 03 05:54:36.680682 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:36.677926 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" event={"ID":"5732cb6f-524c-4389-8fe2-f81885322ae5","Type":"ContainerDied","Data":"2331d1819ab747057808128e9cb00b4554cac81b2a29a6a548e0f970256ed234"} Nov 03 05:54:37.189039 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:37.188200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:37.189039 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:37.188281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:37.189039 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.188466 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:37.189039 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.188530 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret podName:d73ab4e1-b560-420b-8216-6272bd184f39 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:41.188511286 +0000 UTC m=+10.168660908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret") pod "global-pull-secret-syncer-vspq2" (UID: "d73ab4e1-b560-420b-8216-6272bd184f39") : object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:37.189039 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.188942 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:37.189039 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.188993 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:54:41.18897587 +0000 UTC m=+10.169125490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:37.289192 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:37.289149 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:37.289387 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.289368 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 05:54:37.289470 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.289396 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 05:54:37.289470 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.289411 2581 projected.go:196] Error preparing data for projected volume kube-api-access-cwsq8 for pod openshift-network-diagnostics/network-check-target-cp5xp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:37.289573 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.289496 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8 podName:219a457b-c7b8-4a1d-8f14-50a5d56bb9bd nodeName:}" failed. No retries permitted until 2026-11-03 05:54:41.289476446 +0000 UTC m=+10.269626056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwsq8" (UniqueName: "kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8") pod "network-check-target-cp5xp" (UID: "219a457b-c7b8-4a1d-8f14-50a5d56bb9bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:37.516321 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:37.516239 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:37.516523 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.516406 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:37.516623 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:37.516601 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:37.516737 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.516715 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:37.516809 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:37.516793 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:37.516905 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:37.516881 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:38.689215 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:38.688469 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sp2qh" event={"ID":"7502a388-bbcd-4916-8b8d-a4258c58d575","Type":"ContainerStarted","Data":"43afbc8946ced93d36cdf38f6dfdd7e8443948f34a84508f2981cc1c0b879a2c"} Nov 03 05:54:38.733442 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:38.733378 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sp2qh" podStartSLOduration=2.594947647 podStartE2EDuration="6.733357099s" totalImagesPullingTime="4.138409452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:38.732857044 +0000 UTC m=+7.713006675" watchObservedRunningTime="2026-11-03 05:54:38.733357099 +0000 UTC m=+7.713506729" Nov 03 05:54:39.223062 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:39.223030 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:39.224042 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:39.223799 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:39.516764 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:39.515863 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:39.516764 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:39.516030 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:39.516764 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:39.516446 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:39.516764 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:39.516568 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:39.516764 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:39.516624 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:39.516764 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:39.516693 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:39.697040 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:39.696978 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vvkcc" event={"ID":"1eb02066-c20a-45b3-885a-0e9834b1c998","Type":"ContainerStarted","Data":"2a5d02549dfc80dbab4652ce54811f9cd30836472c4243a7cc64e86cba146167"} Nov 03 05:54:39.697546 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:39.697061 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:39.697942 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:39.697829 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:54:39.797621 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:39.797314 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vvkcc" podStartSLOduration=2.597662916 podStartE2EDuration="7.797299876s" totalImagesPullingTime="5.19963696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:39.749073168 +0000 UTC m=+8.729222798" watchObservedRunningTime="2026-11-03 05:54:39.797299876 +0000 UTC m=+8.777449570" Nov 03 05:54:40.703133 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:40.703088 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" event={"ID":"893a8c11-60f6-4736-9971-10520459a1d0","Type":"ContainerStarted","Data":"117f5a1bd5cb97dd674ca889a1cf39ec52a54a6d970f8cc9deefd0031c02feab"} Nov 03 05:54:41.226628 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.226590 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:41.226787 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.226671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:41.226855 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.226803 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:41.226908 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.226869 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret podName:d73ab4e1-b560-420b-8216-6272bd184f39 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:49.226848211 +0000 UTC m=+18.206997843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret") pod "global-pull-secret-syncer-vspq2" (UID: "d73ab4e1-b560-420b-8216-6272bd184f39") : object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:41.227308 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.227285 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:41.227436 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.227349 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:54:49.227331463 +0000 UTC m=+18.207481092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:41.327352 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.327315 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:41.327615 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.327594 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 05:54:41.327716 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.327625 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 05:54:41.327716 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.327640 2581 projected.go:196] Error preparing data for projected volume kube-api-access-cwsq8 for pod openshift-network-diagnostics/network-check-target-cp5xp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:41.327716 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.327701 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8 podName:219a457b-c7b8-4a1d-8f14-50a5d56bb9bd nodeName:}" failed. No retries permitted until 2026-11-03 05:54:49.327680436 +0000 UTC m=+18.307830059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwsq8" (UniqueName: "kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8") pod "network-check-target-cp5xp" (UID: "219a457b-c7b8-4a1d-8f14-50a5d56bb9bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:41.360160 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.360132 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 05:54:41.510087 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.509289 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T05:54:41.360157319Z","UUID":"d7ac9c5a-e043-43e6-a24e-91e22845700c","Handler":null,"Name":"","Endpoint":""} Nov 03 05:54:41.511346 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.510930 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 05:54:41.511346 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.510957 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 05:54:41.517384 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.517362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:41.517753 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.517576 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:41.517753 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.517614 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:41.519323 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.517982 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:41.519478 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.518083 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:41.519608 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:41.518147 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:41.711986 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:41.711948 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" event={"ID":"893a8c11-60f6-4736-9971-10520459a1d0","Type":"ContainerStarted","Data":"09191375bffc45c75b7f3dec51f9ff8aaef7a75d20a9acca65568a7ca056269f"} Nov 03 05:54:42.715989 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:42.715711 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-99n4t" event={"ID":"36beafc8-4aee-448b-8dfd-64f89637aa9f","Type":"ContainerStarted","Data":"d36c5ac3959e0b257bf07c36088fe24e057e97e3d27c7e4e1586ec3401fba793"} Nov 03 05:54:42.717854 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:42.717821 2581 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="f6cc04c515aaaf642c4b79f84099c72d16af04656eecddc6b64cdf746da308ae" exitCode=0 Nov 03 05:54:42.718089 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:42.718065 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" event={"ID":"5732cb6f-524c-4389-8fe2-f81885322ae5","Type":"ContainerDied","Data":"f6cc04c515aaaf642c4b79f84099c72d16af04656eecddc6b64cdf746da308ae"} Nov 03 05:54:42.720102 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:42.720079 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w4bpx" event={"ID":"2f174941-0801-466a-8cef-c2da4faa916f","Type":"ContainerStarted","Data":"5d9a59a872853b28ff54f851403f38ac1d4e96280c8ecabf19ce6129d3627ddf"} Nov 03 05:54:42.722566 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:42.722541 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" event={"ID":"893a8c11-60f6-4736-9971-10520459a1d0","Type":"ContainerStarted","Data":"66d78f84dd004ff75e0cf01024b218ae57bef85b0e237a98ddda26b56d7abaee"} Nov 03 05:54:42.810747 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:42.810693 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-99n4t" podStartSLOduration=2.314909007 podStartE2EDuration="10.81067922s" totalImagesPullingTime="8.495770213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:42.758553761 +0000 UTC m=+11.738703387" watchObservedRunningTime="2026-11-03 05:54:42.81067922 +0000 UTC m=+11.790828842" Nov 03 05:54:42.810901 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:42.810780 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-w4bpx" podStartSLOduration=2.49412299 podStartE2EDuration="10.810771748s" totalImagesPullingTime="8.316648758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:42.81021126 +0000 UTC m=+11.790360885" watchObservedRunningTime="2026-11-03 05:54:42.810771748 +0000 UTC m=+11.790921374" Nov 03 05:54:42.903804 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:42.903745 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" podStartSLOduration=2.6338864600000003 podStartE2EDuration="10.903727249s" totalImagesPullingTime="8.269840789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:42.855458067 +0000 UTC m=+11.835607689" watchObservedRunningTime="2026-11-03 05:54:42.903727249 +0000 UTC m=+11.883876874" Nov 03 05:54:43.514408 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:43.513811 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:43.514408 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:43.513817 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:43.514408 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:43.513973 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:43.514408 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:43.513993 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:43.514408 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:43.514186 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:43.514408 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:43.514281 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:43.728494 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:43.728460 2581 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="4f915b2edd5f43c4f26437b9298c05548fadac00bcf366a7feb915adf334da18" exitCode=0 Nov 03 05:54:43.728996 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:43.728504 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" event={"ID":"5732cb6f-524c-4389-8fe2-f81885322ae5","Type":"ContainerDied","Data":"4f915b2edd5f43c4f26437b9298c05548fadac00bcf366a7feb915adf334da18"} Nov 03 05:54:44.732685 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:44.732602 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" event={"ID":"85bfd499-9f29-4d6e-b2de-094d011b4f0b","Type":"ContainerStarted","Data":"39233fd1f637223ab0bf3c249c5c6283bbfa36b69e0099d0a699afc88e3ee8cd"} Nov 03 05:54:44.734791 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:44.734432 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzjql" event={"ID":"2104714e-6924-4133-90b9-7d98135901e8","Type":"ContainerStarted","Data":"dd57332438ec75572029f54875dd287923afcc774bc7f9cbc223f75d91b17281"} Nov 03 05:54:44.776733 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:44.776677 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" podStartSLOduration=2.083558903 podStartE2EDuration="12.776663617s" totalImagesPullingTime="10.693104714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:44.776099107 +0000 UTC m=+13.756248758" watchObservedRunningTime="2026-11-03 05:54:44.776663617 +0000 UTC m=+13.756813242" Nov 03 05:54:44.816373 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:44.816320 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xzjql" podStartSLOduration=4.523248278 podStartE2EDuration="12.816300117s" totalImagesPullingTime="8.293051839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:44.815957951 +0000 UTC m=+13.796107569" watchObservedRunningTime="2026-11-03 05:54:44.816300117 +0000 UTC m=+13.796449741" Nov 03 05:54:45.517203 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:45.517174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:45.517369 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:45.517309 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:45.517369 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:45.517328 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:45.517506 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:45.517405 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:45.517506 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:45.517436 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:45.517597 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:45.517539 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:45.739098 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:45.739059 2581 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="195d0cedb16bbd0aa18b28e8af1ee4a18033cc13583f6e6c14dfca3ca0ab22f2" exitCode=0 Nov 03 05:54:45.739700 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:45.739149 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" event={"ID":"5732cb6f-524c-4389-8fe2-f81885322ae5","Type":"ContainerDied","Data":"195d0cedb16bbd0aa18b28e8af1ee4a18033cc13583f6e6c14dfca3ca0ab22f2"} Nov 03 05:54:47.513745 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:47.513517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:47.514244 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:47.513538 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:47.514244 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:47.513598 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:47.514244 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:47.513876 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:47.514244 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:47.513997 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:47.514244 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:47.514079 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:47.746696 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:47.746628 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" event={"ID":"5732cb6f-524c-4389-8fe2-f81885322ae5","Type":"ContainerStarted","Data":"8e57a6ba76f01c8db94bc7b21b40a639c5b33033198345e331427bb02de6fa95"} Nov 03 05:54:48.750343 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:48.750309 2581 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="8e57a6ba76f01c8db94bc7b21b40a639c5b33033198345e331427bb02de6fa95" exitCode=0 Nov 03 05:54:48.750761 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:48.750366 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" event={"ID":"5732cb6f-524c-4389-8fe2-f81885322ae5","Type":"ContainerDied","Data":"8e57a6ba76f01c8db94bc7b21b40a639c5b33033198345e331427bb02de6fa95"} Nov 03 05:54:49.284362 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.284153 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:49.284362 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.284342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:49.284526 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.284293 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:49.284526 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.284444 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:49.284526 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.284452 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.284437148 +0000 UTC m=+34.264586765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 05:54:49.284526 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.284483 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret podName:d73ab4e1-b560-420b-8216-6272bd184f39 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.284472721 +0000 UTC m=+34.264622337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret") pod "global-pull-secret-syncer-vspq2" (UID: "d73ab4e1-b560-420b-8216-6272bd184f39") : object "kube-system"/"original-pull-secret" not registered Nov 03 05:54:49.384614 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.384582 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:49.384744 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.384712 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 05:54:49.384744 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.384727 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 05:54:49.384744 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.384737 2581 projected.go:196] Error preparing data for projected volume kube-api-access-cwsq8 for pod openshift-network-diagnostics/network-check-target-cp5xp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:49.384857 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.384783 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8 podName:219a457b-c7b8-4a1d-8f14-50a5d56bb9bd nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.384768305 +0000 UTC m=+34.364917910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwsq8" (UniqueName: "kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8") pod "network-check-target-cp5xp" (UID: "219a457b-c7b8-4a1d-8f14-50a5d56bb9bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 05:54:49.513612 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.513577 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:49.513751 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.513686 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:49.513751 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.513691 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:49.513891 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.513778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:49.513891 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.513777 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:49.513891 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:49.513848 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:49.755505 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.755468 2581 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="095019cc6d00ad522b05a58baab581877979711bfad98f14b2ef9e82949e6435" exitCode=0 Nov 03 05:54:49.756005 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.755555 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" event={"ID":"5732cb6f-524c-4389-8fe2-f81885322ae5","Type":"ContainerDied","Data":"095019cc6d00ad522b05a58baab581877979711bfad98f14b2ef9e82949e6435"} Nov 03 05:54:49.758638 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.758619 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" event={"ID":"0b722d0a-df66-409f-a7cf-0a891f5a1d32","Type":"ContainerStarted","Data":"a29428b65d0203366a2ebfe2dd0a0a32dd7e0610a5083a6a2f391241286e2753"} Nov 03 05:54:49.758699 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.758674 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" event={"ID":"0b722d0a-df66-409f-a7cf-0a891f5a1d32","Type":"ContainerStarted","Data":"649c7e7a6a8ed64094850f22fdb814c18dfb37021fa4aebbc961f0fe27d18c4b"} Nov 03 05:54:49.758738 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.758704 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" event={"ID":"0b722d0a-df66-409f-a7cf-0a891f5a1d32","Type":"ContainerStarted","Data":"3e3766a333e285f135bc5fd7257025c1b83b056703d2afcd8b66c123d015c2bb"} Nov 03 05:54:49.758767 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.758738 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" event={"ID":"0b722d0a-df66-409f-a7cf-0a891f5a1d32","Type":"ContainerStarted","Data":"3f34cb6b521f0a5bdcad1ea945faac1cfaf047bb95d48ce3bf770c43da475436"} Nov 03 05:54:49.758794 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.758765 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" event={"ID":"0b722d0a-df66-409f-a7cf-0a891f5a1d32","Type":"ContainerStarted","Data":"7b9b9a0780114ec267dcd6ad97e67f7f17a26cec3ca3b37c72e0e41e75b30fd5"} Nov 03 05:54:49.758822 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:49.758793 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" event={"ID":"0b722d0a-df66-409f-a7cf-0a891f5a1d32","Type":"ContainerStarted","Data":"53a23733256fab5c8da07bd511ca199d1166de9a51eab1ad83090152c412938c"} Nov 03 05:54:50.763176 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:50.763144 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" event={"ID":"5732cb6f-524c-4389-8fe2-f81885322ae5","Type":"ContainerStarted","Data":"9c8c60e1a15062ba43e3fae9745a62c4b456ec573318da9edc55494ea8880d7a"} Nov 03 05:54:50.839501 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:50.839438 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9c7rn" podStartSLOduration=8.097457633 podStartE2EDuration="18.839424981s" totalImagesPullingTime="10.741967348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:50.839068768 +0000 UTC m=+19.819218394" watchObservedRunningTime="2026-11-03 05:54:50.839424981 +0000 UTC m=+19.819574606" Nov 03 05:54:51.515828 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:51.515752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:51.515828 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:51.515789 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:51.515995 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:51.515752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:51.515995 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:51.515854 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:51.515995 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:51.515934 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:51.516124 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:51.516009 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:51.768476 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:51.768370 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" event={"ID":"0b722d0a-df66-409f-a7cf-0a891f5a1d32","Type":"ContainerStarted","Data":"47861705b5ea475163877eeec6198f789cf7eac41d7d85e98065d76b5628b378"} Nov 03 05:54:53.513647 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:53.513410 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:53.514117 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:53.513411 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:53.514117 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:53.513749 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:53.514117 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:53.513454 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:53.514117 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:53.513811 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:53.514117 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:53.513866 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:53.776296 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:53.776262 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" event={"ID":"0b722d0a-df66-409f-a7cf-0a891f5a1d32","Type":"ContainerStarted","Data":"005adce3f92f6ffb6cb40a62933e71941d02954183f34825b601db47b9421faf"} Nov 03 05:54:53.776635 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:53.776605 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:53.791541 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:53.791521 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:53.869572 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:53.869528 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" podStartSLOduration=6.9160009559999995 podStartE2EDuration="21.869515887s" totalImagesPullingTime="14.953514931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:53.856468593 +0000 UTC m=+22.836618218" watchObservedRunningTime="2026-11-03 05:54:53.869515887 +0000 UTC m=+22.849665489" Nov 03 05:54:54.778835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:54.778804 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:54.779274 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:54.778888 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:54.793735 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:54.793708 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:54:55.345579 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:55.345515 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vspq2"] Nov 03 05:54:55.345727 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:55.345604 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:55.345727 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:55.345685 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:55.348342 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:55.348306 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hbdpd"] Nov 03 05:54:55.348492 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:55.348435 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:55.348577 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:55.348555 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:55.348952 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:55.348935 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cp5xp"] Nov 03 05:54:55.349017 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:55.349007 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:55.349101 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:55.349077 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:56.513838 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:56.513805 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:56.514282 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:56.513846 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:56.514282 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:56.513916 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:56.514282 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:56.514024 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:57.513555 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:57.513521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:57.513709 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:57.513663 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:54:58.513378 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:58.513343 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:54:58.513866 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:58.513343 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:54:58.513866 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:58.513497 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:54:58.513866 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:58.513526 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:54:59.513021 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:54:59.512991 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:54:59.513197 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:54:59.513095 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:55:00.513977 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:00.513942 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:00.514499 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:00.514088 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hbdpd" podUID="58637e90-c0b9-44da-91e9-7e1075a2636e" Nov 03 05:55:00.514499 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:00.514135 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:55:00.514499 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:00.514230 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cp5xp" podUID="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" Nov 03 05:55:01.514018 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.513869 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:01.514436 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:01.514072 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vspq2" podUID="d73ab4e1-b560-420b-8216-6272bd184f39" Nov 03 05:55:01.816571 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.816548 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeReady" Nov 03 05:55:01.816716 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.816659 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 05:55:01.871848 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.871817 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bhkn5"] Nov 03 05:55:01.878507 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.878469 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6z5gg"] Nov 03 05:55:01.878663 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.878604 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bhkn5" Nov 03 05:55:01.884909 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.884858 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tnzjq"] Nov 03 05:55:01.885228 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.885184 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6z5gg" Nov 03 05:55:01.888146 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.888127 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf"] Nov 03 05:55:01.888302 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.888246 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tnzjq" Nov 03 05:55:01.891020 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.891000 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7"] Nov 03 05:55:01.891133 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.891119 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:01.894026 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.894004 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-86pnb"] Nov 03 05:55:01.894198 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.894183 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:01.898090 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.898069 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75b97bb597-swr58"] Nov 03 05:55:01.898201 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.898180 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:01.901623 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.901603 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qwvxg"] Nov 03 05:55:01.901718 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.901702 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:01.904608 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.904585 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 05:55:01.907664 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.907647 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 05:55:01.908139 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.908118 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 05:55:01.910217 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.909178 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk"] Nov 03 05:55:01.910217 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.909812 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:01.923063 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.923041 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j"] Nov 03 05:55:01.923300 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.923282 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk" Nov 03 05:55:01.938968 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.938948 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zknxx"] Nov 03 05:55:01.939075 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.939062 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:01.957221 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.957194 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn"] Nov 03 05:55:01.958256 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.957669 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:01.968992 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.968968 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80c48a6b-a95c-4c64-9d89-1ed9d0320908-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:01.969105 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.968998 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-stats-auth\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:01.969105 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969015 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c48a6b-a95c-4c64-9d89-1ed9d0320908-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:01.969105 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969084 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-service-ca-bundle\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:01.969256 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:01.969256 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969201 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fm7p\" (UniqueName: \"kubernetes.io/projected/db236f0a-264d-4498-8dd1-1787e2a3ee95-kube-api-access-5fm7p\") pod \"kas-connection-checker-685895c7c9-bhkn5\" (UID: \"db236f0a-264d-4498-8dd1-1787e2a3ee95\") " pod="kube-system/kas-connection-checker-685895c7c9-bhkn5" Nov 03 05:55:01.969256 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969224 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/56b77a5b-a55e-4849-9789-b9c5a3db4b02-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:01.969414 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969276 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp2g4\" (UniqueName: \"kubernetes.io/projected/054b9d40-fa74-4f73-9db4-70ff0dd43191-kube-api-access-zp2g4\") pod \"kas-connection-checker-685895c7c9-tnzjq\" (UID: \"054b9d40-fa74-4f73-9db4-70ff0dd43191\") " pod="kube-system/kas-connection-checker-685895c7c9-tnzjq" Nov 03 05:55:01.969414 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969296 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:01.969414 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969341 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:01.969414 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969365 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/315726a9-3009-40f6-bf65-6818acd95046-serving-cert\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:01.969612 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5g9x\" (UniqueName: \"kubernetes.io/projected/80c48a6b-a95c-4c64-9d89-1ed9d0320908-kube-api-access-g5g9x\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:01.969612 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969509 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8z96\" (UniqueName: \"kubernetes.io/projected/76c09b47-0ab3-4212-924b-131d88e94ab3-kube-api-access-q8z96\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:01.969612 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969554 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-snapshots\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:01.969612 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969572 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmvr6\" (UniqueName: \"kubernetes.io/projected/315726a9-3009-40f6-bf65-6818acd95046-kube-api-access-vmvr6\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:01.969612 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969589 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-tmp\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:01.969612 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969605 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znfx\" (UniqueName: \"kubernetes.io/projected/9ef729d9-2e44-4c39-a018-5df7a941dcc5-kube-api-access-4znfx\") pod \"kas-connection-checker-685895c7c9-6z5gg\" (UID: \"9ef729d9-2e44-4c39-a018-5df7a941dcc5\") " pod="kube-system/kas-connection-checker-685895c7c9-6z5gg" Nov 03 05:55:01.969906 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969637 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-default-certificate\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:01.969906 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969660 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:01.969906 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.969680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqgfp\" (UniqueName: \"kubernetes.io/projected/56b77a5b-a55e-4849-9789-b9c5a3db4b02-kube-api-access-dqgfp\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:01.971260 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.971236 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67898f7b5f-gmmrp"] Nov 03 05:55:01.971381 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.971340 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn" Nov 03 05:55:01.991542 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.991522 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c96d9b8b6-5hh78"] Nov 03 05:55:01.991640 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:01.991631 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.004769 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.004750 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:02.004862 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.004821 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 05:55:02.004862 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.004830 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 05:55:02.004862 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.004855 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 05:55:02.004982 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.004908 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 05:55:02.004982 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.004942 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 05:55:02.004982 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.004950 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:02.004982 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.004979 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:02.005117 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005003 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:02.005117 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005037 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 05:55:02.005216 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005203 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 05:55:02.005269 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005236 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:02.005269 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005245 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 05:55:02.005269 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005246 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 05:55:02.005269 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005264 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 05:55:02.005450 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005272 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:02.005450 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005329 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 05:55:02.005450 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005358 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:02.005450 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005406 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 05:55:02.005450 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.005434 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 05:55:02.007526 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.007504 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq"] Nov 03 05:55:02.007651 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.007635 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.017801 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.017787 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 05:55:02.018182 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.018166 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:02.019257 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.019242 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 05:55:02.019318 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.019273 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 05:55:02.019318 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.019286 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 05:55:02.019318 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.019296 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 05:55:02.019508 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.019494 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 05:55:02.019557 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.019522 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-97pnm\"" Nov 03 05:55:02.021755 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.021741 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bhkn5"] Nov 03 05:55:02.021793 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.021768 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6j7pt"] Nov 03 05:55:02.021876 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.021864 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.029482 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.029466 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 05:55:02.029482 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.029475 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 05:55:02.029482 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.029465 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 05:55:02.029621 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.029486 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 05:55:02.029621 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.029479 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 05:55:02.029621 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.029479 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 05:55:02.031455 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.031440 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 05:55:02.031964 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.031949 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 05:55:02.032920 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.032903 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 05:55:02.033025 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.033009 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 05:55:02.041671 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.041652 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g9htv"] Nov 03 05:55:02.041789 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.041774 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:02.065928 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.065907 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6z5gg"] Nov 03 05:55:02.066011 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.065952 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tnzjq"] Nov 03 05:55:02.066011 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.065972 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7"] Nov 03 05:55:02.066011 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.065988 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf"] Nov 03 05:55:02.066011 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066002 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-86pnb"] Nov 03 05:55:02.066129 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066020 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qwvxg"] Nov 03 05:55:02.066129 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066040 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75b97bb597-swr58"] Nov 03 05:55:02.066129 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066059 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c96d9b8b6-5hh78"] Nov 03 05:55:02.066129 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066077 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6j7pt"] Nov 03 05:55:02.066129 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066090 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.066129 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066100 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk"] Nov 03 05:55:02.066129 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066120 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j"] Nov 03 05:55:02.066374 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066143 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g9htv"] Nov 03 05:55:02.066374 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066168 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zknxx"] Nov 03 05:55:02.066374 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066186 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn"] Nov 03 05:55:02.066374 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066205 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq"] Nov 03 05:55:02.066374 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.066226 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67898f7b5f-gmmrp"] Nov 03 05:55:02.070789 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.070740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80c48a6b-a95c-4c64-9d89-1ed9d0320908-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:02.070789 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.070771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-stats-auth\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:02.070951 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.070796 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-service-ca-bundle\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.070951 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.070828 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5pnq\" (UniqueName: \"kubernetes.io/projected/1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec-kube-api-access-t5pnq\") pod \"network-check-source-6f78d59f99-n9hcn\" (UID: \"1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn" Nov 03 05:55:02.070951 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.070858 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-installation-pull-secrets\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.070951 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.070888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:02.070951 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.070913 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr54g\" (UniqueName: \"kubernetes.io/projected/2b099592-879f-4313-be96-60d39b44f8a8-kube-api-access-nr54g\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.070951 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.070940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:02.071251 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.071237 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:02.571220521 +0000 UTC m=+31.551370162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : configmap references non-existent config key: service-ca.crt Nov 03 05:55:02.071342 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071264 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkw2l\" (UniqueName: \"kubernetes.io/projected/99c39ff9-f070-4c29-8284-4f98d116faa4-kube-api-access-lkw2l\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:02.071342 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/56b77a5b-a55e-4849-9789-b9c5a3db4b02-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:02.071342 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:02.071501 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.071501 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:02.071501 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/315726a9-3009-40f6-bf65-6818acd95046-serving-cert\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.071501 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071452 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80c48a6b-a95c-4c64-9d89-1ed9d0320908-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:02.071501 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wktj9\" (UniqueName: \"kubernetes.io/projected/42da3e8e-febb-4e95-ae4c-c750c4bd3481-kube-api-access-wktj9\") pod \"volume-data-source-validator-58c9f5f46d-74hnk\" (UID: \"42da3e8e-febb-4e95-ae4c-c750c4bd3481\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk" Nov 03 05:55:02.071695 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071514 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl4ch\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-kube-api-access-gl4ch\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.071695 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.071558 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 05:55:02.072147 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071601 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-service-ca-bundle\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.072867 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.071592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5g9x\" (UniqueName: \"kubernetes.io/projected/80c48a6b-a95c-4c64-9d89-1ed9d0320908-kube-api-access-g5g9x\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:02.072975 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.072873 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:02.571590023 +0000 UTC m=+31.551739646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : secret "router-metrics-certs-default" not found Nov 03 05:55:02.073086 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.073065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8z96\" (UniqueName: \"kubernetes.io/projected/76c09b47-0ab3-4212-924b-131d88e94ab3-kube-api-access-q8z96\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:02.073198 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.073182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b099592-879f-4313-be96-60d39b44f8a8-serving-cert\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.073268 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.073234 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-snapshots\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.073317 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.073268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmvr6\" (UniqueName: \"kubernetes.io/projected/315726a9-3009-40f6-bf65-6818acd95046-kube-api-access-vmvr6\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.073452 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.073369 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/56b77a5b-a55e-4849-9789-b9c5a3db4b02-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:02.073525 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.073460 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-image-registry-private-configuration\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.073525 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.073507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-tmp\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.073618 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.073538 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-config\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.074227 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.074199 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:02.074318 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.074245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgfp\" (UniqueName: \"kubernetes.io/projected/56b77a5b-a55e-4849-9789-b9c5a3db4b02-kube-api-access-dqgfp\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:02.074318 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.074250 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.074446 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.074316 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-trusted-ca\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.074446 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.074356 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-ca-trust-extracted\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.074532 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.074507 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:02.074577 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.074563 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls podName:56b77a5b-a55e-4849-9789-b9c5a3db4b02 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:02.574549131 +0000 UTC m=+31.554698747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vpktf" (UID: "56b77a5b-a55e-4849-9789-b9c5a3db4b02") : secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.074791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c48a6b-a95c-4c64-9d89-1ed9d0320908-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075052 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-tmp\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075175 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075210 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-trusted-ca\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075245 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-snapshots\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-bound-sa-token\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075317 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fm7p\" (UniqueName: \"kubernetes.io/projected/db236f0a-264d-4498-8dd1-1787e2a3ee95-kube-api-access-5fm7p\") pod \"kas-connection-checker-685895c7c9-bhkn5\" (UID: \"db236f0a-264d-4498-8dd1-1787e2a3ee95\") " pod="kube-system/kas-connection-checker-685895c7c9-bhkn5" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075642 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-certificates\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075687 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp2g4\" (UniqueName: \"kubernetes.io/projected/054b9d40-fa74-4f73-9db4-70ff0dd43191-kube-api-access-zp2g4\") pod \"kas-connection-checker-685895c7c9-tnzjq\" (UID: \"054b9d40-fa74-4f73-9db4-70ff0dd43191\") " pod="kube-system/kas-connection-checker-685895c7c9-tnzjq" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075752 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:02.075856 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znfx\" (UniqueName: \"kubernetes.io/projected/9ef729d9-2e44-4c39-a018-5df7a941dcc5-kube-api-access-4znfx\") pod \"kas-connection-checker-685895c7c9-6z5gg\" (UID: \"9ef729d9-2e44-4c39-a018-5df7a941dcc5\") " pod="kube-system/kas-connection-checker-685895c7c9-6z5gg" Nov 03 05:55:02.077758 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.075987 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-default-certificate\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:02.077758 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.076769 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-stats-auth\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:02.077758 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.077201 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c48a6b-a95c-4c64-9d89-1ed9d0320908-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:02.078814 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.078789 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-default-certificate\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:02.080516 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.080496 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/315726a9-3009-40f6-bf65-6818acd95046-serving-cert\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.105350 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.105326 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:02.105350 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.105332 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 05:55:02.105523 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.105378 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 05:55:02.105523 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.105519 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 05:55:02.105614 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.105538 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:02.110879 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.110831 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 05:55:02.111485 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.111459 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 05:55:02.111578 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.111553 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 05:55:02.111578 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.111552 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 05:55:02.111699 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.111679 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwk29\"" Nov 03 05:55:02.111766 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.111747 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xj6m8\"" Nov 03 05:55:02.112598 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.112575 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmvr6\" (UniqueName: \"kubernetes.io/projected/315726a9-3009-40f6-bf65-6818acd95046-kube-api-access-vmvr6\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.113010 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.112993 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8z96\" (UniqueName: \"kubernetes.io/projected/76c09b47-0ab3-4212-924b-131d88e94ab3-kube-api-access-q8z96\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:02.113294 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.113277 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5g9x\" (UniqueName: \"kubernetes.io/projected/80c48a6b-a95c-4c64-9d89-1ed9d0320908-kube-api-access-g5g9x\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:02.113532 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.113508 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znfx\" (UniqueName: \"kubernetes.io/projected/9ef729d9-2e44-4c39-a018-5df7a941dcc5-kube-api-access-4znfx\") pod \"kas-connection-checker-685895c7c9-6z5gg\" (UID: \"9ef729d9-2e44-4c39-a018-5df7a941dcc5\") " pod="kube-system/kas-connection-checker-685895c7c9-6z5gg" Nov 03 05:55:02.113721 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.113704 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgfp\" (UniqueName: \"kubernetes.io/projected/56b77a5b-a55e-4849-9789-b9c5a3db4b02-kube-api-access-dqgfp\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:02.114514 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.114496 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fm7p\" (UniqueName: \"kubernetes.io/projected/db236f0a-264d-4498-8dd1-1787e2a3ee95-kube-api-access-5fm7p\") pod \"kas-connection-checker-685895c7c9-bhkn5\" (UID: \"db236f0a-264d-4498-8dd1-1787e2a3ee95\") " pod="kube-system/kas-connection-checker-685895c7c9-bhkn5" Nov 03 05:55:02.116830 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.116813 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp2g4\" (UniqueName: \"kubernetes.io/projected/054b9d40-fa74-4f73-9db4-70ff0dd43191-kube-api-access-zp2g4\") pod \"kas-connection-checker-685895c7c9-tnzjq\" (UID: \"054b9d40-fa74-4f73-9db4-70ff0dd43191\") " pod="kube-system/kas-connection-checker-685895c7c9-tnzjq" Nov 03 05:55:02.176894 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.176868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-config\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.177011 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.176902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.177011 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.176929 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:02.177099 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.177099 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177041 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pnq\" (UniqueName: \"kubernetes.io/projected/1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec-kube-api-access-t5pnq\") pod \"network-check-source-6f78d59f99-n9hcn\" (UID: \"1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn" Nov 03 05:55:02.177192 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177094 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-installation-pull-secrets\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.177192 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.177115 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:02.177192 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.177128 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67898f7b5f-gmmrp: secret "image-registry-tls" not found Nov 03 05:55:02.177192 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177179 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/059a8188-2f82-4b0d-af82-5598a6e7b50e-cert\") pod \"ingress-canary-6j7pt\" (UID: \"059a8188-2f82-4b0d-af82-5598a6e7b50e\") " pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:02.177379 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.177198 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls podName:f29d1b5f-b1e2-45e0-8b1f-da929d562b71 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:02.677178273 +0000 UTC m=+31.657327890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls") pod "image-registry-67898f7b5f-gmmrp" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71") : secret "image-registry-tls" not found Nov 03 05:55:02.177379 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf7vt\" (UniqueName: \"kubernetes.io/projected/059a8188-2f82-4b0d-af82-5598a6e7b50e-kube-api-access-tf7vt\") pod \"ingress-canary-6j7pt\" (UID: \"059a8188-2f82-4b0d-af82-5598a6e7b50e\") " pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:02.177379 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177280 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.177379 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd3a09f2-9d41-40f6-94f1-6cb00971d950-tmp-dir\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.177602 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr54g\" (UniqueName: \"kubernetes.io/projected/2b099592-879f-4313-be96-60d39b44f8a8-kube-api-access-nr54g\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.177602 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:02.177602 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177490 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkw2l\" (UniqueName: \"kubernetes.io/projected/99c39ff9-f070-4c29-8284-4f98d116faa4-kube-api-access-lkw2l\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:02.177602 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177524 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-certificates\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.177602 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-864s8\" (UniqueName: \"kubernetes.io/projected/bd3a09f2-9d41-40f6-94f1-6cb00971d950-kube-api-access-864s8\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.177602 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.177560 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 05:55:02.177602 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177582 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4tsh\" (UniqueName: \"kubernetes.io/projected/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-kube-api-access-k4tsh\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.177602 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177523 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.177615 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls podName:99c39ff9-f070-4c29-8284-4f98d116faa4 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:02.677597294 +0000 UTC m=+31.657746897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d277j" (UID: "99c39ff9-f070-4c29-8284-4f98d116faa4") : secret "samples-operator-tls" not found Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.177746 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177755 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74a0e103-b720-45e0-8f30-c54e9542b5cf-ca-trust-extracted\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.177778 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert podName:8ba1e446-7be2-4673-9a44-a6c6a5e1be2d nodeName:}" failed. No retries permitted until 2026-11-03 05:55:02.677769375 +0000 UTC m=+31.657918978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zknxx" (UID: "8ba1e446-7be2-4673-9a44-a6c6a5e1be2d") : secret "networking-console-plugin-cert" not found Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177814 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktj9\" (UniqueName: \"kubernetes.io/projected/42da3e8e-febb-4e95-ae4c-c750c4bd3481-kube-api-access-wktj9\") pod \"volume-data-source-validator-58c9f5f46d-74hnk\" (UID: \"42da3e8e-febb-4e95-ae4c-c750c4bd3481\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk" Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177841 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4ch\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-kube-api-access-gl4ch\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b099592-879f-4313-be96-60d39b44f8a8-serving-cert\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-image-registry-private-configuration\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-config\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.177973 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-installation-pull-secrets\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.178506 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.177998 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-trusted-ca\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.178506 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178025 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-ca-trust-extracted\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.178506 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178049 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79lwr\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-kube-api-access-79lwr\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.178506 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a09f2-9d41-40f6-94f1-6cb00971d950-config-volume\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.178506 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178106 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-image-registry-private-configuration\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.178506 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:02.178506 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-trusted-ca\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.178506 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178483 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-ca-trust-extracted\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.178879 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178550 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-bound-sa-token\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.178879 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178671 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-config\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.178879 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-certificates\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.178879 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178780 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-serving-cert\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.178879 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178816 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-trusted-ca\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.178879 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178878 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-bound-sa-token\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.179156 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178943 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:02.179156 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.178995 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-trusted-ca\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.179331 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.179301 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-certificates\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.179440 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.179355 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-trusted-ca\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.179669 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.179649 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-installation-pull-secrets\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.180177 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.180159 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b099592-879f-4313-be96-60d39b44f8a8-serving-cert\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.180803 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.180784 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-image-registry-private-configuration\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.199870 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.199849 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bhkn5" Nov 03 05:55:02.203162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.203140 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkw2l\" (UniqueName: \"kubernetes.io/projected/99c39ff9-f070-4c29-8284-4f98d116faa4-kube-api-access-lkw2l\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:02.203327 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.203309 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pnq\" (UniqueName: \"kubernetes.io/projected/1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec-kube-api-access-t5pnq\") pod \"network-check-source-6f78d59f99-n9hcn\" (UID: \"1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn" Nov 03 05:55:02.208729 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.208711 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr54g\" (UniqueName: \"kubernetes.io/projected/2b099592-879f-4313-be96-60d39b44f8a8-kube-api-access-nr54g\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.211092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.211065 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-bound-sa-token\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.211284 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.211269 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktj9\" (UniqueName: \"kubernetes.io/projected/42da3e8e-febb-4e95-ae4c-c750c4bd3481-kube-api-access-wktj9\") pod \"volume-data-source-validator-58c9f5f46d-74hnk\" (UID: \"42da3e8e-febb-4e95-ae4c-c750c4bd3481\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk" Nov 03 05:55:02.211361 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.211341 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl4ch\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-kube-api-access-gl4ch\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.231281 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.231261 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6z5gg" Nov 03 05:55:02.266008 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.265982 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:02.279360 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-serving-cert\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.279501 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279364 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-trusted-ca\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.279501 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279405 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-bound-sa-token\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.279501 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-config\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.279595 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.279595 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/059a8188-2f82-4b0d-af82-5598a6e7b50e-cert\") pod \"ingress-canary-6j7pt\" (UID: \"059a8188-2f82-4b0d-af82-5598a6e7b50e\") " pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:02.279595 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.279585 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 05:55:02.279715 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279597 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf7vt\" (UniqueName: \"kubernetes.io/projected/059a8188-2f82-4b0d-af82-5598a6e7b50e-kube-api-access-tf7vt\") pod \"ingress-canary-6j7pt\" (UID: \"059a8188-2f82-4b0d-af82-5598a6e7b50e\") " pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:02.279715 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.279715 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.279635 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls podName:bd3a09f2-9d41-40f6-94f1-6cb00971d950 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:02.779619307 +0000 UTC m=+31.759768941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls") pod "dns-default-g9htv" (UID: "bd3a09f2-9d41-40f6-94f1-6cb00971d950") : secret "dns-default-metrics-tls" not found Nov 03 05:55:02.279715 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd3a09f2-9d41-40f6-94f1-6cb00971d950-tmp-dir\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.279715 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.279665 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:02.279715 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.279677 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c96d9b8b6-5hh78: secret "image-registry-tls" not found Nov 03 05:55:02.279715 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.279714 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls podName:74a0e103-b720-45e0-8f30-c54e9542b5cf nodeName:}" failed. No retries permitted until 2026-11-03 05:55:02.779704361 +0000 UTC m=+31.759853966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls") pod "image-registry-7c96d9b8b6-5hh78" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf") : secret "image-registry-tls" not found Nov 03 05:55:02.279986 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-certificates\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.279986 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279773 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-864s8\" (UniqueName: \"kubernetes.io/projected/bd3a09f2-9d41-40f6-94f1-6cb00971d950-kube-api-access-864s8\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.279986 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279802 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4tsh\" (UniqueName: \"kubernetes.io/projected/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-kube-api-access-k4tsh\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.279986 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.279873 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74a0e103-b720-45e0-8f30-c54e9542b5cf-ca-trust-extracted\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.281540 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.280379 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd3a09f2-9d41-40f6-94f1-6cb00971d950-tmp-dir\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.281540 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.280538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-installation-pull-secrets\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.281540 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.280594 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79lwr\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-kube-api-access-79lwr\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.281540 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.280622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a09f2-9d41-40f6-94f1-6cb00971d950-config-volume\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.281540 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.280656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-image-registry-private-configuration\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.281540 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.280908 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74a0e103-b720-45e0-8f30-c54e9542b5cf-ca-trust-extracted\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.281540 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.281165 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-config\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.281540 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.281437 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-trusted-ca\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.281540 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.281490 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-certificates\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.282009 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.281731 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a09f2-9d41-40f6-94f1-6cb00971d950-config-volume\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.282344 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.282325 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/059a8188-2f82-4b0d-af82-5598a6e7b50e-cert\") pod \"ingress-canary-6j7pt\" (UID: \"059a8188-2f82-4b0d-af82-5598a6e7b50e\") " pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:02.282344 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.282320 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-serving-cert\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.283741 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.283724 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-image-registry-private-configuration\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.283871 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.283854 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-installation-pull-secrets\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.314254 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.314209 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-bound-sa-token\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.314621 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.314596 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf7vt\" (UniqueName: \"kubernetes.io/projected/059a8188-2f82-4b0d-af82-5598a6e7b50e-kube-api-access-tf7vt\") pod \"ingress-canary-6j7pt\" (UID: \"059a8188-2f82-4b0d-af82-5598a6e7b50e\") " pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:02.315162 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.315115 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4tsh\" (UniqueName: \"kubernetes.io/projected/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-kube-api-access-k4tsh\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.316366 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.315927 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-864s8\" (UniqueName: \"kubernetes.io/projected/bd3a09f2-9d41-40f6-94f1-6cb00971d950-kube-api-access-864s8\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.317297 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.317263 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79lwr\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-kube-api-access-79lwr\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.332583 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.332555 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:02.357500 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.357300 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk" Nov 03 05:55:02.380733 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.380706 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:02.411421 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.411373 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tnzjq" Nov 03 05:55:02.460652 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.459892 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn" Nov 03 05:55:02.496728 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.496702 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:02.510754 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.510730 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:02.514041 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.513977 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:02.514348 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.514324 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:55:02.573303 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.573130 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bhkn5"] Nov 03 05:55:02.585894 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.580918 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6z5gg"] Nov 03 05:55:02.585894 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.583585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:02.585894 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.583655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:02.585894 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.583741 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:03.583723314 +0000 UTC m=+32.563872927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : configmap references non-existent config key: service-ca.crt Nov 03 05:55:02.585894 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.584056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:02.585894 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.583812 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 05:55:02.585894 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.584218 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:03.584206371 +0000 UTC m=+32.564355998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : secret "router-metrics-certs-default" not found Nov 03 05:55:02.585894 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.584145 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:02.585894 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.584253 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls podName:56b77a5b-a55e-4849-9789-b9c5a3db4b02 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:03.584245397 +0000 UTC m=+32.564395029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vpktf" (UID: "56b77a5b-a55e-4849-9789-b9c5a3db4b02") : secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:02.591894 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.591850 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 05:55:02.592050 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.592018 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xvf66\"" Nov 03 05:55:02.592288 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.592229 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mlps4\"" Nov 03 05:55:02.602895 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.602861 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7"] Nov 03 05:55:02.652637 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.652602 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-86pnb"] Nov 03 05:55:02.656801 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:55:02.656778 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315726a9_3009_40f6_bf65_6818acd95046.slice/crio-6e9a399c0d95841d6e79f3805b1ce884a2f866ce675185df36cc0cf6d62737b6 WatchSource:0}: Error finding container 6e9a399c0d95841d6e79f3805b1ce884a2f866ce675185df36cc0cf6d62737b6: Status 404 returned error can't find the container with id 6e9a399c0d95841d6e79f3805b1ce884a2f866ce675185df36cc0cf6d62737b6 Nov 03 05:55:02.686075 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.685680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:02.686075 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.685750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:02.686075 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.685828 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 05:55:02.686075 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.685869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:02.686075 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.685895 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls podName:99c39ff9-f070-4c29-8284-4f98d116faa4 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:03.685874801 +0000 UTC m=+32.666024420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d277j" (UID: "99c39ff9-f070-4c29-8284-4f98d116faa4") : secret "samples-operator-tls" not found Nov 03 05:55:02.686075 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.685968 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 05:55:02.686075 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.685986 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:02.686075 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.686000 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67898f7b5f-gmmrp: secret "image-registry-tls" not found Nov 03 05:55:02.686075 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.686001 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert podName:8ba1e446-7be2-4673-9a44-a6c6a5e1be2d nodeName:}" failed. No retries permitted until 2026-11-03 05:55:03.68599286 +0000 UTC m=+32.666142484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zknxx" (UID: "8ba1e446-7be2-4673-9a44-a6c6a5e1be2d") : secret "networking-console-plugin-cert" not found Nov 03 05:55:02.686075 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.686059 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls podName:f29d1b5f-b1e2-45e0-8b1f-da929d562b71 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:03.686043731 +0000 UTC m=+32.666193338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls") pod "image-registry-67898f7b5f-gmmrp" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71") : secret "image-registry-tls" not found Nov 03 05:55:02.691821 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.691759 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk"] Nov 03 05:55:02.707270 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:55:02.707225 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42da3e8e_febb_4e95_ae4c_c750c4bd3481.slice/crio-189c597ef65ce3c5390ace18f63184a962739e8fdb474b724c16c94e528534d8 WatchSource:0}: Error finding container 189c597ef65ce3c5390ace18f63184a962739e8fdb474b724c16c94e528534d8: Status 404 returned error can't find the container with id 189c597ef65ce3c5390ace18f63184a962739e8fdb474b724c16c94e528534d8 Nov 03 05:55:02.708693 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.708666 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qwvxg"] Nov 03 05:55:02.713056 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:55:02.713034 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b099592_879f_4313_be96_60d39b44f8a8.slice/crio-e1fcf9b67df930930c3aa224ef49c55b3fd82d8b92b15764d2fb6e7a0890cfff WatchSource:0}: Error finding container e1fcf9b67df930930c3aa224ef49c55b3fd82d8b92b15764d2fb6e7a0890cfff: Status 404 returned error can't find the container with id e1fcf9b67df930930c3aa224ef49c55b3fd82d8b92b15764d2fb6e7a0890cfff Nov 03 05:55:02.741618 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.741144 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tnzjq"] Nov 03 05:55:02.750456 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:55:02.750425 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054b9d40_fa74_4f73_9db4_70ff0dd43191.slice/crio-25e3df992bc69f8ac07b758425f8e2a70002aea8f351e9d65368ae1b2ad3fcc1 WatchSource:0}: Error finding container 25e3df992bc69f8ac07b758425f8e2a70002aea8f351e9d65368ae1b2ad3fcc1: Status 404 returned error can't find the container with id 25e3df992bc69f8ac07b758425f8e2a70002aea8f351e9d65368ae1b2ad3fcc1 Nov 03 05:55:02.787046 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.787015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:02.787142 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.787065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:02.787422 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.787382 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:02.787422 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.787423 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c96d9b8b6-5hh78: secret "image-registry-tls" not found Nov 03 05:55:02.787569 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.787483 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls podName:74a0e103-b720-45e0-8f30-c54e9542b5cf nodeName:}" failed. No retries permitted until 2026-11-03 05:55:03.787458691 +0000 UTC m=+32.767608310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls") pod "image-registry-7c96d9b8b6-5hh78" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf") : secret "image-registry-tls" not found Nov 03 05:55:02.787569 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.787557 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 05:55:02.787679 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:02.787592 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls podName:bd3a09f2-9d41-40f6-94f1-6cb00971d950 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:03.78758094 +0000 UTC m=+32.767730558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls") pod "dns-default-g9htv" (UID: "bd3a09f2-9d41-40f6-94f1-6cb00971d950") : secret "dns-default-metrics-tls" not found Nov 03 05:55:02.790632 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.790605 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn"] Nov 03 05:55:02.795275 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.795222 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" event={"ID":"2b099592-879f-4313-be96-60d39b44f8a8","Type":"ContainerStarted","Data":"e1fcf9b67df930930c3aa224ef49c55b3fd82d8b92b15764d2fb6e7a0890cfff"} Nov 03 05:55:02.795545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:55:02.795518 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a38e8c6_468e_4a7f_8804_fde7ea3bb3ec.slice/crio-c0f1d323aff656dfbaf8d35db010d71881b756f47b15ea263f2040ca0188a2fb WatchSource:0}: Error finding container c0f1d323aff656dfbaf8d35db010d71881b756f47b15ea263f2040ca0188a2fb: Status 404 returned error can't find the container with id c0f1d323aff656dfbaf8d35db010d71881b756f47b15ea263f2040ca0188a2fb Nov 03 05:55:02.796770 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.796743 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" event={"ID":"80c48a6b-a95c-4c64-9d89-1ed9d0320908","Type":"ContainerStarted","Data":"bf72b91612df2fa8a4c1f58f81062a6ea5c9c64f16e05ac77b5bd370024b970f"} Nov 03 05:55:02.800436 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.800003 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6z5gg" event={"ID":"9ef729d9-2e44-4c39-a018-5df7a941dcc5","Type":"ContainerStarted","Data":"266c806013757285d08026fb42f6bb60aa18e4015e982a5db815d9812f60d7b6"} Nov 03 05:55:02.800436 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.800049 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6z5gg" event={"ID":"9ef729d9-2e44-4c39-a018-5df7a941dcc5","Type":"ContainerStarted","Data":"24334de0cccdb5ad2f73dd53e8955dccdfeb342780e7de20d73aaf49c830c0fa"} Nov 03 05:55:02.801604 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.801577 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk" event={"ID":"42da3e8e-febb-4e95-ae4c-c750c4bd3481","Type":"ContainerStarted","Data":"189c597ef65ce3c5390ace18f63184a962739e8fdb474b724c16c94e528534d8"} Nov 03 05:55:02.802668 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.802641 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-86pnb" event={"ID":"315726a9-3009-40f6-bf65-6818acd95046","Type":"ContainerStarted","Data":"6e9a399c0d95841d6e79f3805b1ce884a2f866ce675185df36cc0cf6d62737b6"} Nov 03 05:55:02.803810 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.803786 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tnzjq" event={"ID":"054b9d40-fa74-4f73-9db4-70ff0dd43191","Type":"ContainerStarted","Data":"25e3df992bc69f8ac07b758425f8e2a70002aea8f351e9d65368ae1b2ad3fcc1"} Nov 03 05:55:02.805192 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.805169 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bhkn5" event={"ID":"db236f0a-264d-4498-8dd1-1787e2a3ee95","Type":"ContainerStarted","Data":"2105c9c556f7939e4c9519e87848b017001a9caadbcadf470c77984df8a6d89f"} Nov 03 05:55:02.805283 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.805209 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bhkn5" event={"ID":"db236f0a-264d-4498-8dd1-1787e2a3ee95","Type":"ContainerStarted","Data":"683bb386ac9f4af2f1ac192455504ae55d1098799acc53eacbf3cd9d40c31469"} Nov 03 05:55:02.836559 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.836525 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq"] Nov 03 05:55:02.839459 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:55:02.839436 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f4a643_0b2d_45ca_9ab7_3ad2ae287a8f.slice/crio-cc273797706cc875480faf0a6abdfb6868d5972e4e8d767eb4523c02b7f71edc WatchSource:0}: Error finding container cc273797706cc875480faf0a6abdfb6868d5972e4e8d767eb4523c02b7f71edc: Status 404 returned error can't find the container with id cc273797706cc875480faf0a6abdfb6868d5972e4e8d767eb4523c02b7f71edc Nov 03 05:55:02.841200 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.841165 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6z5gg" podStartSLOduration=384.841154105 podStartE2EDuration="6m24.841154105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:02.841023428 +0000 UTC m=+31.821173053" watchObservedRunningTime="2026-11-03 05:55:02.841154105 +0000 UTC m=+31.821303723" Nov 03 05:55:02.869820 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.869794 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6j7pt"] Nov 03 05:55:02.872999 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:55:02.872968 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059a8188_2f82_4b0d_af82_5598a6e7b50e.slice/crio-1081d6240a8fba18caf5647b6ada31caccf4aeb3a962329ccd5675689030b54e WatchSource:0}: Error finding container 1081d6240a8fba18caf5647b6ada31caccf4aeb3a962329ccd5675689030b54e: Status 404 returned error can't find the container with id 1081d6240a8fba18caf5647b6ada31caccf4aeb3a962329ccd5675689030b54e Nov 03 05:55:02.881984 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:02.881940 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bhkn5" podStartSLOduration=384.881928261 podStartE2EDuration="6m24.881928261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:02.881488485 +0000 UTC m=+31.861638111" watchObservedRunningTime="2026-11-03 05:55:02.881928261 +0000 UTC m=+31.862077885" Nov 03 05:55:03.516852 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.516816 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:03.532635 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.532608 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rwdxr\"" Nov 03 05:55:03.532744 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.532701 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 05:55:03.596199 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.596164 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:03.596371 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.596237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:03.596371 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.596323 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:03.596557 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.596484 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:03.596557 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.596547 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls podName:56b77a5b-a55e-4849-9789-b9c5a3db4b02 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.596528376 +0000 UTC m=+34.576678010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vpktf" (UID: "56b77a5b-a55e-4849-9789-b9c5a3db4b02") : secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:03.596977 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.596954 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.596938877 +0000 UTC m=+34.577088494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : configmap references non-existent config key: service-ca.crt Nov 03 05:55:03.597065 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.597028 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 05:55:03.597065 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.597060 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.597049506 +0000 UTC m=+34.577199124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : secret "router-metrics-certs-default" not found Nov 03 05:55:03.697693 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.697656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:03.697866 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.697785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:03.697866 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.697857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:03.698011 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.697994 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 05:55:03.698076 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.698064 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls podName:99c39ff9-f070-4c29-8284-4f98d116faa4 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.698043311 +0000 UTC m=+34.678192935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d277j" (UID: "99c39ff9-f070-4c29-8284-4f98d116faa4") : secret "samples-operator-tls" not found Nov 03 05:55:03.698569 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.698548 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 05:55:03.698678 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.698603 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert podName:8ba1e446-7be2-4673-9a44-a6c6a5e1be2d nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.698588583 +0000 UTC m=+34.678738200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zknxx" (UID: "8ba1e446-7be2-4673-9a44-a6c6a5e1be2d") : secret "networking-console-plugin-cert" not found Nov 03 05:55:03.698678 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.698668 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:03.698678 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.698679 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67898f7b5f-gmmrp: secret "image-registry-tls" not found Nov 03 05:55:03.698832 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.698714 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls podName:f29d1b5f-b1e2-45e0-8b1f-da929d562b71 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.698703125 +0000 UTC m=+34.678852742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls") pod "image-registry-67898f7b5f-gmmrp" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71") : secret "image-registry-tls" not found Nov 03 05:55:03.799334 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.798567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:03.799334 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.798611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:03.799334 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.798770 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:03.799334 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.798785 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c96d9b8b6-5hh78: secret "image-registry-tls" not found Nov 03 05:55:03.799334 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.798842 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls podName:74a0e103-b720-45e0-8f30-c54e9542b5cf nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.798824747 +0000 UTC m=+34.778974367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls") pod "image-registry-7c96d9b8b6-5hh78" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf") : secret "image-registry-tls" not found Nov 03 05:55:03.799334 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.799255 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 05:55:03.799334 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:03.799300 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls podName:bd3a09f2-9d41-40f6-94f1-6cb00971d950 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.79928686 +0000 UTC m=+34.779436476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls") pod "dns-default-g9htv" (UID: "bd3a09f2-9d41-40f6-94f1-6cb00971d950") : secret "dns-default-metrics-tls" not found Nov 03 05:55:03.811664 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.811361 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tnzjq" event={"ID":"054b9d40-fa74-4f73-9db4-70ff0dd43191","Type":"ContainerStarted","Data":"54180852de47bdd84d30d1e7b9aa149801b1ff11a4de2179be789efcce3b72b5"} Nov 03 05:55:03.814359 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.814331 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6j7pt" event={"ID":"059a8188-2f82-4b0d-af82-5598a6e7b50e","Type":"ContainerStarted","Data":"1081d6240a8fba18caf5647b6ada31caccf4aeb3a962329ccd5675689030b54e"} Nov 03 05:55:03.817575 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.817453 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" event={"ID":"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f","Type":"ContainerStarted","Data":"cc273797706cc875480faf0a6abdfb6868d5972e4e8d767eb4523c02b7f71edc"} Nov 03 05:55:03.821464 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:03.820786 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn" event={"ID":"1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec","Type":"ContainerStarted","Data":"c0f1d323aff656dfbaf8d35db010d71881b756f47b15ea263f2040ca0188a2fb"} Nov 03 05:55:05.316421 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.316366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:05.316903 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.316575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:05.317191 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.317169 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 05:55:05.317267 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.317237 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:55:37.3172224 +0000 UTC m=+66.297372017 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : secret "metrics-daemon-secret" not found Nov 03 05:55:05.328477 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.328446 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:05.353801 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.353770 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:05.424832 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.424801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:55:05.434092 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.434065 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:55:05.578537 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.578450 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:55:05.627664 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.627032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:05.627664 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.627093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:05.627664 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.627166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:05.627664 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.627312 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:09.627297954 +0000 UTC m=+38.607447569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : configmap references non-existent config key: service-ca.crt Nov 03 05:55:05.627664 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.627578 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 05:55:05.627664 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.627617 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:05.627664 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.627651 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:09.627633159 +0000 UTC m=+38.607782776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : secret "router-metrics-certs-default" not found Nov 03 05:55:05.627664 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.627672 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls podName:56b77a5b-a55e-4849-9789-b9c5a3db4b02 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:09.627661805 +0000 UTC m=+38.607811421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vpktf" (UID: "56b77a5b-a55e-4849-9789-b9c5a3db4b02") : secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:05.715901 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.715399 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tnzjq" podStartSLOduration=387.715365348 podStartE2EDuration="6m27.715365348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:03.863523097 +0000 UTC m=+32.843672723" watchObservedRunningTime="2026-11-03 05:55:05.715365348 +0000 UTC m=+34.695514973" Nov 03 05:55:05.715901 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.715626 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vspq2"] Nov 03 05:55:05.728148 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.727711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:05.728148 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.727774 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:05.728148 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.727860 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:05.728148 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.727958 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:05.728148 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.727968 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67898f7b5f-gmmrp: secret "image-registry-tls" not found Nov 03 05:55:05.728148 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.728002 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls podName:f29d1b5f-b1e2-45e0-8b1f-da929d562b71 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:09.727990701 +0000 UTC m=+38.708140304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls") pod "image-registry-67898f7b5f-gmmrp" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71") : secret "image-registry-tls" not found Nov 03 05:55:05.728148 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.728050 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 05:55:05.728148 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.728071 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls podName:99c39ff9-f070-4c29-8284-4f98d116faa4 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:09.728064055 +0000 UTC m=+38.708213658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d277j" (UID: "99c39ff9-f070-4c29-8284-4f98d116faa4") : secret "samples-operator-tls" not found Nov 03 05:55:05.728148 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.728106 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 05:55:05.728148 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.728124 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert podName:8ba1e446-7be2-4673-9a44-a6c6a5e1be2d nodeName:}" failed. No retries permitted until 2026-11-03 05:55:09.728118734 +0000 UTC m=+38.708268336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zknxx" (UID: "8ba1e446-7be2-4673-9a44-a6c6a5e1be2d") : secret "networking-console-plugin-cert" not found Nov 03 05:55:05.829366 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.828303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:05.829366 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.828356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:05.829366 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.828671 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:05.829366 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.828688 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c96d9b8b6-5hh78: secret "image-registry-tls" not found Nov 03 05:55:05.829366 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.828744 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls podName:74a0e103-b720-45e0-8f30-c54e9542b5cf nodeName:}" failed. No retries permitted until 2026-11-03 05:55:09.828726367 +0000 UTC m=+38.808875975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls") pod "image-registry-7c96d9b8b6-5hh78" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf") : secret "image-registry-tls" not found Nov 03 05:55:05.829366 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.829224 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 05:55:05.829366 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:05.829277 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls podName:bd3a09f2-9d41-40f6-94f1-6cb00971d950 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:09.829262054 +0000 UTC m=+38.809411659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls") pod "dns-default-g9htv" (UID: "bd3a09f2-9d41-40f6-94f1-6cb00971d950") : secret "dns-default-metrics-tls" not found Nov 03 05:55:05.830451 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.830225 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk" event={"ID":"42da3e8e-febb-4e95-ae4c-c750c4bd3481","Type":"ContainerStarted","Data":"e68f7a1bd8a1ce7cc29a95cd6bd4c5affc3ecd6cbf47f23d364e84a631389228"} Nov 03 05:55:05.833104 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.833026 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vspq2" event={"ID":"d73ab4e1-b560-420b-8216-6272bd184f39","Type":"ContainerStarted","Data":"e32bf93d77ff76fa461e71ab334be2d22802595e294984e161f19b8ab63e28e5"} Nov 03 05:55:05.872493 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.872442 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk" podStartSLOduration=356.581072268 podStartE2EDuration="5m58.87242729s" totalImagesPullingTime="2.291355022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:05.871260411 +0000 UTC m=+34.851410042" watchObservedRunningTime="2026-11-03 05:55:05.87242729 +0000 UTC m=+34.852576935" Nov 03 05:55:05.953604 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:05.953571 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cp5xp"] Nov 03 05:55:05.961545 ip-10-0-141-67 kubenswrapper[2581]: W1103 05:55:05.961515 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219a457b_c7b8_4a1d_8f14_50a5d56bb9bd.slice/crio-5ea405697ad2752b6132fc12e652c621b186fac158e520481546aa51e18a8d70 WatchSource:0}: Error finding container 5ea405697ad2752b6132fc12e652c621b186fac158e520481546aa51e18a8d70: Status 404 returned error can't find the container with id 5ea405697ad2752b6132fc12e652c621b186fac158e520481546aa51e18a8d70 Nov 03 05:55:06.838738 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:06.838701 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cp5xp" event={"ID":"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd","Type":"ContainerStarted","Data":"5ea405697ad2752b6132fc12e652c621b186fac158e520481546aa51e18a8d70"} Nov 03 05:55:07.846089 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:07.845219 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/0.log" Nov 03 05:55:07.846089 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:07.845260 2581 generic.go:356] "Generic (PLEG): container finished" podID="2b099592-879f-4313-be96-60d39b44f8a8" containerID="291f37a54066642b9db32e8846cdb339ac44e1ab841d54fbb83dc1a22a10ca4e" exitCode=255 Nov 03 05:55:07.846089 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:07.845340 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" event={"ID":"2b099592-879f-4313-be96-60d39b44f8a8","Type":"ContainerDied","Data":"291f37a54066642b9db32e8846cdb339ac44e1ab841d54fbb83dc1a22a10ca4e"} Nov 03 05:55:07.846089 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:07.845666 2581 scope.go:122] "RemoveContainer" containerID="291f37a54066642b9db32e8846cdb339ac44e1ab841d54fbb83dc1a22a10ca4e" Nov 03 05:55:07.857216 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:07.856435 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" event={"ID":"80c48a6b-a95c-4c64-9d89-1ed9d0320908","Type":"ContainerStarted","Data":"f2cc64f408500b0480ba2548f6d6b1af6098e971ae45f6d13bc561078c2065b0"} Nov 03 05:55:07.872533 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:07.872106 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-86pnb" event={"ID":"315726a9-3009-40f6-bf65-6818acd95046","Type":"ContainerStarted","Data":"5d615acf6c9415b0697ea23a73d415cacb33a017e38ad366737b36c5ce63b5f0"} Nov 03 05:55:07.934003 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:07.933948 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-86pnb" podStartSLOduration=356.196721501 podStartE2EDuration="6m0.933934421s" totalImagesPullingTime="4.73721292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:07.933551813 +0000 UTC m=+36.913701438" watchObservedRunningTime="2026-11-03 05:55:07.933934421 +0000 UTC m=+36.914084046" Nov 03 05:55:07.978235 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:07.977727 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" podStartSLOduration=356.167346633 podStartE2EDuration="6m0.977709745s" totalImagesPullingTime="4.810363112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:07.975965777 +0000 UTC m=+36.956115402" watchObservedRunningTime="2026-11-03 05:55:07.977709745 +0000 UTC m=+36.957859371" Nov 03 05:55:08.294943 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.294862 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tqt2f"] Nov 03 05:55:08.320367 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.320335 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.323835 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.323806 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tqt2f"] Nov 03 05:55:08.336295 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.336273 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 05:55:08.336458 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.336305 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j9vz5\"" Nov 03 05:55:08.337155 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.337136 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 05:55:08.458433 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.458075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.458433 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.458121 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7hfp\" (UniqueName: \"kubernetes.io/projected/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-api-access-k7hfp\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.458433 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.458194 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.458433 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.458242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08a92738-2cd0-4eed-9e83-c01baed110c2-crio-socket\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.458433 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.458294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08a92738-2cd0-4eed-9e83-c01baed110c2-data-volume\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.560569 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.560535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08a92738-2cd0-4eed-9e83-c01baed110c2-data-volume\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.560749 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.560624 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.560749 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.560658 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7hfp\" (UniqueName: \"kubernetes.io/projected/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-api-access-k7hfp\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.561165 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.560864 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.561165 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.560932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08a92738-2cd0-4eed-9e83-c01baed110c2-crio-socket\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.561165 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:08.560955 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 05:55:08.561165 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:08.561016 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:09.06099842 +0000 UTC m=+38.041148025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : configmap "kube-rbac-proxy" not found Nov 03 05:55:08.561165 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.561059 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08a92738-2cd0-4eed-9e83-c01baed110c2-crio-socket\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.561165 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:08.561150 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 05:55:08.561543 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:08.561187 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:09.061175077 +0000 UTC m=+38.041324685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : secret "insights-runtime-extractor-tls" not found Nov 03 05:55:08.561543 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.561189 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08a92738-2cd0-4eed-9e83-c01baed110c2-data-volume\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.589466 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.589407 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7hfp\" (UniqueName: \"kubernetes.io/projected/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-api-access-k7hfp\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:08.878157 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.878076 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6j7pt" event={"ID":"059a8188-2f82-4b0d-af82-5598a6e7b50e","Type":"ContainerStarted","Data":"c3c51a5e5d879cfb986ce781f99168d3cc6345149f945b6101d9cfbf088090fb"} Nov 03 05:55:08.880169 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.880072 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" event={"ID":"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f","Type":"ContainerStarted","Data":"cadfecc8bda7de97d34adf474edaa53fca1cb15309a5415eb5e1a87ddda409e4"} Nov 03 05:55:08.881872 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.881844 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/1.log" Nov 03 05:55:08.882438 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.882413 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/0.log" Nov 03 05:55:08.882537 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.882458 2581 generic.go:356] "Generic (PLEG): container finished" podID="2b099592-879f-4313-be96-60d39b44f8a8" containerID="a4debb86934e6ea1da15236ab15df42b19e3ca5cef0906b0ce4d4abfd1f63885" exitCode=255 Nov 03 05:55:08.883330 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.882662 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" event={"ID":"2b099592-879f-4313-be96-60d39b44f8a8","Type":"ContainerDied","Data":"a4debb86934e6ea1da15236ab15df42b19e3ca5cef0906b0ce4d4abfd1f63885"} Nov 03 05:55:08.883330 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.882714 2581 scope.go:122] "RemoveContainer" containerID="291f37a54066642b9db32e8846cdb339ac44e1ab841d54fbb83dc1a22a10ca4e" Nov 03 05:55:08.883330 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.883013 2581 scope.go:122] "RemoveContainer" containerID="a4debb86934e6ea1da15236ab15df42b19e3ca5cef0906b0ce4d4abfd1f63885" Nov 03 05:55:08.883330 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:08.883158 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qwvxg_openshift-console-operator(2b099592-879f-4313-be96-60d39b44f8a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" podUID="2b099592-879f-4313-be96-60d39b44f8a8" Nov 03 05:55:08.920265 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:08.920211 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6j7pt" podStartSLOduration=2.426174547 podStartE2EDuration="7.920192149s" totalImagesPullingTime="5.494017602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:08.919242002 +0000 UTC m=+37.899391627" watchObservedRunningTime="2026-11-03 05:55:08.920192149 +0000 UTC m=+37.900341776" Nov 03 05:55:09.025731 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.025674 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" podStartSLOduration=357.087574681 podStartE2EDuration="6m2.025662114s" totalImagesPullingTime="4.938087433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:09.024690395 +0000 UTC m=+38.004840020" watchObservedRunningTime="2026-11-03 05:55:09.025662114 +0000 UTC m=+38.005811739" Nov 03 05:55:09.068065 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.068029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:09.068232 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.068181 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 05:55:09.068322 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.068252 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:10.068232315 +0000 UTC m=+39.048381924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : secret "insights-runtime-extractor-tls" not found Nov 03 05:55:09.068322 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.068247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:09.068322 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.068253 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 05:55:09.068536 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.068358 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:10.068344984 +0000 UTC m=+39.048494588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : configmap "kube-rbac-proxy" not found Nov 03 05:55:09.675126 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.675082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:09.675306 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.675188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:09.675306 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.675264 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 05:55:09.675433 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.675310 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:09.675433 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.675324 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:09.675433 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.675356 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.675333867 +0000 UTC m=+46.655483485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : secret "router-metrics-certs-default" not found Nov 03 05:55:09.675433 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.675426 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls podName:56b77a5b-a55e-4849-9789-b9c5a3db4b02 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.67540819 +0000 UTC m=+46.655557795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vpktf" (UID: "56b77a5b-a55e-4849-9789-b9c5a3db4b02") : secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:09.675637 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.675460 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.675453793 +0000 UTC m=+46.655603396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : configmap references non-existent config key: service-ca.crt Nov 03 05:55:09.776887 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.776818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:09.776887 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.776877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:09.777041 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.776912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:09.777041 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.776957 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:09.777041 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.776976 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67898f7b5f-gmmrp: secret "image-registry-tls" not found Nov 03 05:55:09.777041 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.777010 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 05:55:09.777041 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.777028 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls podName:f29d1b5f-b1e2-45e0-8b1f-da929d562b71 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.777011471 +0000 UTC m=+46.757161091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls") pod "image-registry-67898f7b5f-gmmrp" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71") : secret "image-registry-tls" not found Nov 03 05:55:09.777191 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.777048 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert podName:8ba1e446-7be2-4673-9a44-a6c6a5e1be2d nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.777038072 +0000 UTC m=+46.757187675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zknxx" (UID: "8ba1e446-7be2-4673-9a44-a6c6a5e1be2d") : secret "networking-console-plugin-cert" not found Nov 03 05:55:09.777191 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.777011 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 05:55:09.777191 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.777074 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls podName:99c39ff9-f070-4c29-8284-4f98d116faa4 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.777065083 +0000 UTC m=+46.757214692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d277j" (UID: "99c39ff9-f070-4c29-8284-4f98d116faa4") : secret "samples-operator-tls" not found Nov 03 05:55:09.877495 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.877445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:09.877495 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.877495 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:09.877774 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.877749 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:09.877774 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.877770 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c96d9b8b6-5hh78: secret "image-registry-tls" not found Nov 03 05:55:09.877937 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.877823 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls podName:74a0e103-b720-45e0-8f30-c54e9542b5cf nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.877810511 +0000 UTC m=+46.857960134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls") pod "image-registry-7c96d9b8b6-5hh78" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf") : secret "image-registry-tls" not found Nov 03 05:55:09.877937 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.877842 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 05:55:09.877937 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.877907 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls podName:bd3a09f2-9d41-40f6-94f1-6cb00971d950 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.877887719 +0000 UTC m=+46.858037337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls") pod "dns-default-g9htv" (UID: "bd3a09f2-9d41-40f6-94f1-6cb00971d950") : secret "dns-default-metrics-tls" not found Nov 03 05:55:09.888220 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.888189 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn" event={"ID":"1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec","Type":"ContainerStarted","Data":"51940a278f7ace974729221afd078801c60081f894b5b3ea47c41a5029c430c1"} Nov 03 05:55:09.889961 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.889935 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cp5xp" event={"ID":"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd","Type":"ContainerStarted","Data":"a260d53f9acdc684900adea732115fbc3684128f1e524dac7caaa96f1c249eab"} Nov 03 05:55:09.890369 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.890354 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:55:09.892004 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.891987 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/1.log" Nov 03 05:55:09.892596 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.892582 2581 scope.go:122] "RemoveContainer" containerID="a4debb86934e6ea1da15236ab15df42b19e3ca5cef0906b0ce4d4abfd1f63885" Nov 03 05:55:09.892735 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:09.892716 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qwvxg_openshift-console-operator(2b099592-879f-4313-be96-60d39b44f8a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" podUID="2b099592-879f-4313-be96-60d39b44f8a8" Nov 03 05:55:09.931755 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:09.931705 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn" podStartSLOduration=353.256864284 podStartE2EDuration="5m59.931687959s" totalImagesPullingTime="6.674823675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:09.929836518 +0000 UTC m=+38.909986142" watchObservedRunningTime="2026-11-03 05:55:09.931687959 +0000 UTC m=+38.911837585" Nov 03 05:55:10.018847 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:10.018792 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cp5xp" podStartSLOduration=34.52146817 podStartE2EDuration="38.018775322s" totalImagesPullingTime="3.497307152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:10.018068192 +0000 UTC m=+38.998217817" watchObservedRunningTime="2026-11-03 05:55:10.018775322 +0000 UTC m=+38.998924947" Nov 03 05:55:10.080087 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:10.080052 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:10.080274 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:10.080180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:10.080274 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:10.080214 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 05:55:10.080274 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:10.080247 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 05:55:10.080455 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:10.080281 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:12.080266815 +0000 UTC m=+41.060416436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : secret "insights-runtime-extractor-tls" not found Nov 03 05:55:10.080455 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:10.080300 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:12.080290664 +0000 UTC m=+41.060440267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : configmap "kube-rbac-proxy" not found Nov 03 05:55:12.096324 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:12.096245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:12.096324 ip-10-0-141-67 kubenswrapper[2581]: I1103 05:55:12.096316 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:12.096761 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:12.096382 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 05:55:12.096761 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:12.096420 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 05:55:12.096761 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:12.096468 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:16.096451749 +0000 UTC m=+45.076601368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : configmap "kube-rbac-proxy" not found Nov 03 05:55:12.096761 ip-10-0-141-67 kubenswrapper[2581]: E1103 05:55:12.096484 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:16.096476579 +0000 UTC m=+45.076626182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : secret "insights-runtime-extractor-tls" not found Nov 03 05:55:12.123274 ip-10-0-141-67 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 05:55:12.136773 ip-10-0-141-67 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 05:55:12.136935 ip-10-0-141-67 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 05:55:12.136981 ip-10-0-141-67 systemd[1]: kubelet.service: Consumed 2.890s CPU time. Nov 03 05:55:12.148063 ip-10-0-141-67 systemd[1]: Starting Kubernetes Kubelet... Nov 03 05:55:12.247349 ip-10-0-141-67 kubenswrapper[4855]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 05:55:12.247349 ip-10-0-141-67 kubenswrapper[4855]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 05:55:12.247349 ip-10-0-141-67 kubenswrapper[4855]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 05:55:12.247349 ip-10-0-141-67 kubenswrapper[4855]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 05:55:12.249423 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249327 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:55:12.249423 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249421 4855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:55:12.249423 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249425 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:55:12.249423 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249428 4855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249432 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249435 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249438 4855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249441 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249447 4855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249450 4855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249455 4855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249459 4855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249463 4855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249466 4855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249468 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249471 4855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249474 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249477 4855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249480 4855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249482 4855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249485 4855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249488 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:55:12.249576 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249492 4855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249495 4855 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249499 4855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249501 4855 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249504 4855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249517 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249521 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249524 4855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249527 4855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249529 4855 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249532 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249535 4855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249538 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249541 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249543 4855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249546 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249549 4855 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249552 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249554 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249557 4855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:55:12.250077 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249560 4855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249562 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249565 4855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249567 4855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249570 4855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249573 4855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249577 4855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249582 4855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249585 4855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249588 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249591 4855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249594 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249597 4855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249599 4855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249603 4855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249605 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249608 4855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249611 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249614 4855 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:55:12.250573 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249616 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249619 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249622 4855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249624 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249627 4855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249629 4855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249632 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249635 4855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249638 4855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249642 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249644 4855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249647 4855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249649 4855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249652 4855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249655 4855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249657 4855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249660 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249663 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249665 4855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249668 4855 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:55:12.251027 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249671 4855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249673 4855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249676 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249679 4855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249682 4855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249685 4855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249687 4855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249690 4855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249693 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249696 4855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249699 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249702 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249704 4855 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249707 4855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249709 4855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249712 4855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249715 4855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249717 4855 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249720 4855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249723 4855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:55:12.251538 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249726 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249729 4855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249732 4855 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249735 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.249738 4855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250675 4855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250684 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250687 4855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250691 4855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250694 4855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250697 4855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250701 4855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250703 4855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250706 4855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250709 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250711 4855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250714 4855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250717 4855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250719 4855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250722 4855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:55:12.252024 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250725 4855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250728 4855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250731 4855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250733 4855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250736 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250739 4855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250742 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250744 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250747 4855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250749 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250752 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250754 4855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250757 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250760 4855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250762 4855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250766 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250769 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250772 4855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250774 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:55:12.252527 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250777 4855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250779 4855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250782 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250785 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250787 4855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250790 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250792 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250795 4855 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250797 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250800 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250803 4855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250805 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250808 4855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250810 4855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250813 4855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250815 4855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250818 4855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250820 4855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250823 4855 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250826 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:55:12.253000 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250828 4855 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250831 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250833 4855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250836 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250838 4855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250844 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250846 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250850 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250852 4855 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250858 4855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250861 4855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250864 4855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250867 4855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250869 4855 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250874 4855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250877 4855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250880 4855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250884 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250886 4855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250889 4855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:55:12.253697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250892 4855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250895 4855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250900 4855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250902 4855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250905 4855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250908 4855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250910 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250913 4855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250915 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250918 4855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250920 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250923 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250925 4855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250928 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250931 4855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250933 4855 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250936 4855 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250940 4855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250943 4855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:55:12.254218 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250946 4855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250949 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250951 4855 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250954 4855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250957 4855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250960 4855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250963 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250965 4855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250968 4855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250971 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250973 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250976 4855 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.250979 4855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251533 4855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251540 4855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251543 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251548 4855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251552 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:55:12.254697 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251555 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251558 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251561 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251563 4855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251566 4855 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251569 4855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251572 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251574 4855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251577 4855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251580 4855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251583 4855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251585 4855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251589 4855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251592 4855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251594 4855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251597 4855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251600 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251603 4855 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251606 4855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251608 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:55:12.255147 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251611 4855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251613 4855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251616 4855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251618 4855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251621 4855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251624 4855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251626 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251629 4855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251631 4855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251634 4855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251636 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251639 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251641 4855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251644 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251647 4855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251649 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251652 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251655 4855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251657 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:55:12.255673 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251660 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251662 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251665 4855 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251668 4855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251670 4855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251673 4855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251675 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251678 4855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251680 4855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251683 4855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251685 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251688 4855 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251690 4855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251693 4855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251696 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251700 4855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251703 4855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251706 4855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251709 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251711 4855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:55:12.256131 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251714 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251717 4855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251719 4855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251722 4855 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251724 4855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251727 4855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251730 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251733 4855 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251735 4855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251738 4855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251741 4855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251743 4855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251747 4855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251751 4855 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251754 4855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251757 4855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251760 4855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251763 4855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251766 4855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251768 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:55:12.256939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251771 4855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251773 4855 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251776 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251779 4855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251781 4855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251784 4855 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251787 4855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251791 4855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251793 4855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251796 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251798 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251801 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251804 4855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251807 4855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251810 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251813 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251815 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251819 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251821 4855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:55:12.257768 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251824 4855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251827 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.251829 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251916 4855 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251924 4855 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251930 4855 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251935 4855 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251940 4855 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251943 4855 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251947 4855 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251954 4855 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251957 4855 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251960 4855 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251964 4855 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251967 4855 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251970 4855 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251973 4855 flags.go:64] FLAG: --cgroup-root="" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251976 4855 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251980 4855 flags.go:64] FLAG: --client-ca-file="" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251983 4855 flags.go:64] FLAG: --cloud-provider="external" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251988 4855 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251992 4855 flags.go:64] FLAG: --cluster-domain="" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251994 4855 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.251998 4855 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 05:55:12.258613 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252001 4855 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252004 4855 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252008 4855 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252011 4855 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252014 4855 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252018 4855 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252021 4855 flags.go:64] FLAG: --contention-profiling="false" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252024 4855 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252027 4855 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252031 4855 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252034 4855 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252038 4855 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252041 4855 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252044 4855 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252047 4855 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252051 4855 flags.go:64] FLAG: --enable-server="true" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252054 4855 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252059 4855 flags.go:64] FLAG: --event-burst="100" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252063 4855 flags.go:64] FLAG: --event-qps="50" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252066 4855 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252070 4855 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252072 4855 flags.go:64] FLAG: --eviction-hard="" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252077 4855 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252080 4855 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252083 4855 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 05:55:12.259725 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252087 4855 flags.go:64] FLAG: --eviction-soft="" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252090 4855 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252093 4855 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252099 4855 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252102 4855 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252105 4855 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252108 4855 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252111 4855 flags.go:64] FLAG: --feature-gates="" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252115 4855 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252119 4855 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252122 4855 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252125 4855 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252129 4855 flags.go:64] FLAG: --healthz-port="10248" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252132 4855 flags.go:64] FLAG: --help="false" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252135 4855 flags.go:64] FLAG: --hostname-override="ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252138 4855 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252141 4855 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252144 4855 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252148 4855 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252151 4855 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252154 4855 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252157 4855 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252160 4855 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 05:55:12.260842 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252163 4855 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252166 4855 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252170 4855 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252173 4855 flags.go:64] FLAG: --kube-reserved="" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252176 4855 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252179 4855 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252183 4855 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252186 4855 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252189 4855 flags.go:64] FLAG: --lock-file="" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252192 4855 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252195 4855 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252198 4855 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252205 4855 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252208 4855 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252211 4855 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252214 4855 flags.go:64] FLAG: --logging-format="text" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252217 4855 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252221 4855 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252224 4855 flags.go:64] FLAG: --manifest-url="" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252226 4855 flags.go:64] FLAG: --manifest-url-header="" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252231 4855 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252234 4855 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252238 4855 flags.go:64] FLAG: --max-pods="110" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252241 4855 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252244 4855 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 05:55:12.261843 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252248 4855 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252251 4855 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252254 4855 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252257 4855 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252260 4855 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252267 4855 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252270 4855 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252273 4855 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252276 4855 flags.go:64] FLAG: --pod-cidr="" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252280 4855 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252283 4855 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252286 4855 flags.go:64] FLAG: --pods-per-core="0" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252289 4855 flags.go:64] FLAG: --port="10250" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252292 4855 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252295 4855 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09a764ebb1f190fa9" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252299 4855 flags.go:64] FLAG: --qos-reserved="" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252301 4855 flags.go:64] FLAG: --read-only-port="10255" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252304 4855 flags.go:64] FLAG: --register-node="true" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252308 4855 flags.go:64] FLAG: --register-with-taints="" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252313 4855 flags.go:64] FLAG: --registry-burst="10" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252316 4855 flags.go:64] FLAG: --registry-qps="5" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252319 4855 flags.go:64] FLAG: --reserved-cpus="" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252322 4855 flags.go:64] FLAG: --reserved-memory="" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252326 4855 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252330 4855 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252333 4855 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 05:55:12.262912 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252336 4855 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252338 4855 flags.go:64] FLAG: --runonce="false" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252341 4855 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252344 4855 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252347 4855 flags.go:64] FLAG: --seccomp-default="false" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252350 4855 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252353 4855 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252356 4855 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252360 4855 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252363 4855 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252366 4855 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252369 4855 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252371 4855 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252374 4855 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252377 4855 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252382 4855 flags.go:64] FLAG: --system-cgroups="" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252398 4855 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252404 4855 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252407 4855 flags.go:64] FLAG: --tls-cert-file="" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252410 4855 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252414 4855 flags.go:64] FLAG: --tls-min-version="" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252417 4855 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252420 4855 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252423 4855 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252426 4855 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 05:55:12.264130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252430 4855 flags.go:64] FLAG: --v="2" Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252434 4855 flags.go:64] FLAG: --version="false" Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252439 4855 flags.go:64] FLAG: --vmodule="" Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252443 4855 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252446 4855 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252575 4855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252579 4855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252583 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252586 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252589 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252592 4855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252595 4855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252598 4855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252600 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252603 4855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252605 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252608 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252611 4855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252613 4855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252616 4855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:55:12.265201 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252618 4855 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252621 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252625 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252628 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252631 4855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252634 4855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252636 4855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252639 4855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252642 4855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252644 4855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252648 4855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252652 4855 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252656 4855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252659 4855 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252661 4855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252665 4855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252667 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252670 4855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252673 4855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:55:12.266049 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252675 4855 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252678 4855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252681 4855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252683 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252686 4855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252688 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252692 4855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252696 4855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252699 4855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252702 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252705 4855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252708 4855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252710 4855 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252713 4855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252716 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252720 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252723 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252726 4855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252728 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:55:12.266714 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252731 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252733 4855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252736 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252738 4855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252741 4855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252743 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252747 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252750 4855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252753 4855 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252756 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252759 4855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252761 4855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252764 4855 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252767 4855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252770 4855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252772 4855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252775 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252778 4855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252780 4855 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252783 4855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:55:12.267484 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252785 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252788 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252791 4855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252793 4855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252796 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252799 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252802 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252804 4855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252809 4855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252812 4855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252814 4855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252817 4855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252819 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252822 4855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252824 4855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252827 4855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252830 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252832 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252836 4855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:55:12.268092 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252839 4855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252841 4855 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252844 4855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252847 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252850 4855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252852 4855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252855 4855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252858 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252860 4855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252862 4855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252865 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252868 4855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252870 4855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252873 4855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.252898 4855 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252931 4855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:55:12.268569 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252936 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252939 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252942 4855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252945 4855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252947 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252952 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252954 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252957 4855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252960 4855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252962 4855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252965 4855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252967 4855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252970 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252973 4855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252975 4855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252978 4855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252984 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252986 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252989 4855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:55:12.269043 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252991 4855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252994 4855 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.252997 4855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253000 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253003 4855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253006 4855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253008 4855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253011 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253014 4855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253016 4855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253019 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253022 4855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253025 4855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253028 4855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253030 4855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253033 4855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253035 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253038 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253042 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253044 4855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:55:12.269630 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253047 4855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253050 4855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253052 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253055 4855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253057 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253060 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253063 4855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253065 4855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253068 4855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253073 4855 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253075 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253078 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253082 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253085 4855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253087 4855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253090 4855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253093 4855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253096 4855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253098 4855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:55:12.270148 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253102 4855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253106 4855 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253109 4855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253112 4855 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253114 4855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253117 4855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253120 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253122 4855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253125 4855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253127 4855 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253130 4855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253134 4855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253136 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253139 4855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253141 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253145 4855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253149 4855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253151 4855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253154 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:55:12.270648 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253157 4855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253159 4855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253162 4855 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253166 4855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253169 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253171 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253175 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253178 4855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253180 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253183 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253186 4855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253189 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253192 4855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253194 4855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253197 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253199 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253203 4855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253205 4855 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253209 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253211 4855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:55:12.271112 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253214 4855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253216 4855 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253219 4855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253222 4855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253226 4855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253228 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253231 4855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.253234 4855 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.259078 4855 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.259110 4855 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.259144 4855 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.259153 4855 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259246 4855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259253 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259259 4855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259264 4855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259268 4855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259273 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259278 4855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259284 4855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:55:12.271621 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259290 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259296 4855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259300 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259305 4855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259309 4855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259314 4855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259319 4855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259324 4855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259328 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259333 4855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259337 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259341 4855 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259346 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259351 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259355 4855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259360 4855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259365 4855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259369 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259374 4855 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259379 4855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:55:12.272126 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259383 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259403 4855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259408 4855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259413 4855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259417 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259424 4855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259431 4855 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259437 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259442 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259447 4855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259451 4855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259456 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259461 4855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259465 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259470 4855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259475 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259480 4855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259485 4855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259505 4855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:55:12.272725 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259511 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259515 4855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259520 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259525 4855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259529 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259534 4855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259539 4855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259543 4855 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259548 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259552 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259557 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259561 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259566 4855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259570 4855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259575 4855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259579 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259583 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259588 4855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259592 4855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259596 4855 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:55:12.273177 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259601 4855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259606 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259610 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259615 4855 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259619 4855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259624 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259628 4855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259632 4855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259636 4855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259641 4855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259645 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259650 4855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259656 4855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259661 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259665 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259670 4855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259674 4855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259678 4855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259683 4855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259687 4855 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:55:12.273703 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259692 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259697 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259701 4855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259705 4855 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259710 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259714 4855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259719 4855 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259723 4855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259727 4855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259730 4855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259735 4855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259739 4855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259743 4855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259747 4855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259751 4855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259755 4855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259760 4855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259764 4855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:55:12.274198 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.259769 4855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.259813 4855 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260056 4855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260064 4855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260069 4855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260074 4855 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260078 4855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260084 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260088 4855 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260092 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260097 4855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260101 4855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260106 4855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260111 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260115 4855 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260120 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:55:12.274818 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260124 4855 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260128 4855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260133 4855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260137 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260142 4855 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260147 4855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260151 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260155 4855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260160 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260164 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260168 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260172 4855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260177 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260182 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260186 4855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260190 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260195 4855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260199 4855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260204 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260209 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:55:12.275435 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260213 4855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260218 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260223 4855 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260227 4855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260232 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260237 4855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260241 4855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260245 4855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260250 4855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260254 4855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260259 4855 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260263 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260267 4855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260272 4855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260276 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260280 4855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260285 4855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260289 4855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260294 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260298 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:55:12.276182 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260302 4855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260306 4855 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260311 4855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260315 4855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260320 4855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260324 4855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260329 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260333 4855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260338 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260342 4855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260346 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260350 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260354 4855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260359 4855 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260365 4855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260371 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260376 4855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260384 4855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:55:12.276792 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260405 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260410 4855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260415 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260419 4855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260424 4855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260428 4855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260432 4855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260436 4855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260441 4855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260445 4855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260449 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260453 4855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260458 4855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260463 4855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260467 4855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260472 4855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260476 4855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260481 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260485 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:55:12.277358 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260489 4855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260493 4855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260497 4855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260502 4855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260506 4855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260510 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260514 4855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260519 4855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260523 4855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260527 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260531 4855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260536 4855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260541 4855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260547 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260551 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:55:12.277939 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.260590 4855 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260644 4855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260651 4855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260655 4855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260660 4855 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260664 4855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260669 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260673 4855 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260678 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260682 4855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260686 4855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260691 4855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260696 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260700 4855 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260704 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260708 4855 feature_gate.go:386] unrecognized feature gate: Example Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260713 4855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260717 4855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260721 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260726 4855 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 05:55:12.278431 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260730 4855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260735 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260739 4855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260744 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260748 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260752 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260756 4855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260761 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260766 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260770 4855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260775 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260779 4855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260787 4855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260791 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260796 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260800 4855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260805 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260809 4855 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260813 4855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260818 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 05:55:12.279186 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260822 4855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260826 4855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260831 4855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260835 4855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260839 4855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260843 4855 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260847 4855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260851 4855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260855 4855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260859 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260863 4855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260870 4855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260874 4855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260879 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260883 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260889 4855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260894 4855 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260898 4855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260903 4855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260907 4855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 05:55:12.279851 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260911 4855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260916 4855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260922 4855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260927 4855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260934 4855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260938 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260943 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260947 4855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260951 4855 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260958 4855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260964 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260969 4855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260976 4855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260982 4855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260987 4855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260991 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.260996 4855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261000 4855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 05:55:12.280549 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261004 4855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261009 4855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261013 4855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261018 4855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261022 4855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261028 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261032 4855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261037 4855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261041 4855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261045 4855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261050 4855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261055 4855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261059 4855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261063 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261067 4855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261072 4855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261076 4855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261080 4855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261086 4855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 05:55:12.281086 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261092 4855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261097 4855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261101 4855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261105 4855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261110 4855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261114 4855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261121 4855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261125 4855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261129 4855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:12.261134 4855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.261217 4855 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.262804 4855 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.262932 4855 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.263208 4855 server.go:1014] "Starting client certificate rotation" Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.263345 4855 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.263412 4855 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 05:49:32 +0000 UTC" deadline="2028-07-29 09:30:49.79235289 +0000 UTC" Nov 03 05:55:12.281918 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.263451 4855 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15219h35m37.52890429s" Nov 03 05:55:12.282596 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.263967 4855 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 05:55:12.282596 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.265894 4855 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 05:55:12.282596 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.267263 4855 log.go:25] "Validated CRI v1 runtime API" Nov 03 05:55:12.282596 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.268978 4855 log.go:25] "Validated CRI v1 image API" Nov 03 05:55:12.282596 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.269742 4855 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 05:55:12.282596 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.271686 4855 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9c6b24d3-750a-44d5-bd68-7a6d348f221c:/dev/nvme0n1p3 c68664e9-c913-479b-b5cc-616a813a84e1:/dev/nvme0n1p4] Nov 03 05:55:12.282897 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.271705 4855 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/089f26e348dc1b9a0fa160f005b1cfafce0b63c24e1e9fa4597d30eace4648dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/089f26e348dc1b9a0fa160f005b1cfafce0b63c24e1e9fa4597d30eace4648dc/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1081d6240a8fba18caf5647b6ada31caccf4aeb3a962329ccd5675689030b54e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1081d6240a8fba18caf5647b6ada31caccf4aeb3a962329ccd5675689030b54e/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/189c597ef65ce3c5390ace18f63184a962739e8fdb474b724c16c94e528534d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/189c597ef65ce3c5390ace18f63184a962739e8fdb474b724c16c94e528534d8/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24334de0cccdb5ad2f73dd53e8955dccdfeb342780e7de20d73aaf49c830c0fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24334de0cccdb5ad2f73dd53e8955dccdfeb342780e7de20d73aaf49c830c0fa/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25e3df992bc69f8ac07b758425f8e2a70002aea8f351e9d65368ae1b2ad3fcc1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25e3df992bc69f8ac07b758425f8e2a70002aea8f351e9d65368ae1b2ad3fcc1/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5624d691f94f024b3a87e1b0882e0b03e3ee89535ab7e8ac3a47a52ebf962e2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5624d691f94f024b3a87e1b0882e0b03e3ee89535ab7e8ac3a47a52ebf962e2e/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ea405697ad2752b6132fc12e652c621b186fac158e520481546aa51e18a8d70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ea405697ad2752b6132fc12e652c621b186fac158e520481546aa51e18a8d70/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ec17c2466431329949305a0981ed793552f6226ff216836e2d1be2e6e904ce1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ec17c2466431329949305a0981ed793552f6226ff216836e2d1be2e6e904ce1/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/649c95861450533aae74aa2e6b4a98dd10706551a5f6c510e57802b6561ad0f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/649c95861450533aae74aa2e6b4a98dd10706551a5f6c510e57802b6561ad0f5/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/683bb386ac9f4af2f1ac192455504ae55d1098799acc53eacbf3cd9d40c31469/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/683bb386ac9f4af2f1ac192455504ae55d1098799acc53eacbf3cd9d40c31469/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6dd0b7a3516914b9370f372e6e54c49b972ea78d5a7ccc80fa468db31d7edd9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6dd0b7a3516914b9370f372e6e54c49b972ea78d5a7ccc80fa468db31d7edd9d/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e9a399c0d95841d6e79f3805b1ce884a2f866ce675185df36cc0cf6d62737b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e9a399c0d95841d6e79f3805b1ce884a2f866ce675185df36cc0cf6d62737b6/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/718c715c7de938642fa7a115a4af073f8ac12f8aba6373708322979065e4f003/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/718c715c7de938642fa7a115a4af073f8ac12f8aba6373708322979065e4f003/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b549c16578dd87e9bdf0b6ac99e5583d0b7c04dd077ab74fcde8d5a2de7f045c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b549c16578dd87e9bdf0b6ac99e5583d0b7c04dd077ab74fcde8d5a2de7f045c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf72b91612df2fa8a4c1f58f81062a6ea5c9c64f16e05ac77b5bd370024b970f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf72b91612df2fa8a4c1f58f81062a6ea5c9c64f16e05ac77b5bd370024b970f/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0f1d323aff656dfbaf8d35db010d71881b756f47b15ea263f2040ca0188a2fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0f1d323aff656dfbaf8d35db010d71881b756f47b15ea263f2040ca0188a2fb/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5f9a19056772b54b900cbd373eca0b0ce02f0484ad370a914463ec0753e00dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5f9a19056772b54b900cbd373eca0b0ce02f0484ad370a914463ec0753e00dd/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc273797706cc875480faf0a6abdfb6868d5972e4e8d767eb4523c02b7f71edc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc273797706cc875480faf0a6abdfb6868d5972e4e8d767eb4523c02b7f71edc/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1fcf9b67df930930c3aa224ef49c55b3fd82d8b92b15764d2fb6e7a0890cfff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1fcf9b67df930930c3aa224ef49c55b3fd82d8b92b15764d2fb6e7a0890cfff/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e32bf93d77ff76fa461e71ab334be2d22802595e294984e161f19b8ab63e28e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e32bf93d77ff76fa461e71ab334be2d22802595e294984e161f19b8ab63e28e5/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee9b430004788822fe9777f5b33e99c1c93ea7f922c7ddac81d34c7980535505/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee9b430004788822fe9777f5b33e99c1c93ea7f922c7ddac81d34c7980535505/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7e9d06a2a1bd320cf8e4d6451881ffe3de535e8906f94c326b01ca451574213/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7e9d06a2a1bd320cf8e4d6451881ffe3de535e8906f94c326b01ca451574213/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/054b9d40-fa74-4f73-9db4-70ff0dd43191/volumes/kubernetes.io~projected/kube-api-access-zp2g4:{mountpoint:/var/lib/kubelet/pods/054b9d40-fa74-4f73-9db4-70ff0dd43191/volumes/kubernetes.io~projected/kube-api-access-zp2g4 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/059a8188-2f82-4b0d-af82-5598a6e7b50e/volumes/kubernetes.io~projected/kube-api-access-tf7vt:{mountpoint:/var/lib/kubelet/pods/059a8188-2f82-4b0d-af82-5598a6e7b50e/volumes/kubernetes.io~projected/kube-api-access-tf7vt major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/059a8188-2f82-4b0d-af82-5598a6e7b50e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/059a8188-2f82-4b0d-af82-5598a6e7b50e/volumes/kubernetes.io~secret/cert major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08a92738-2cd0-4eed-9e83-c01baed110c2/volumes/kubernetes.io~projected/kube-api-access-k7hfp:{mountpoint:/var/lib/kubelet/pods/08a92738-2cd0-4eed-9e83-c01baed110c2/volumes/kubernetes.io~projected/kube-api-access-k7hfp major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b722d0a-df66-409f-a7cf-0a891f5a1d32/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0b722d0a-df66-409f-a7cf-0a891f5a1d32/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b722d0a-df66-409f-a7cf-0a891f5a1d32/volumes/kubernetes.io~projected/kube-api-access-89mmd:{mountpoint:/var/lib/kubelet/pods/0b722d0a-df66-409f-a7cf-0a891f5a1d32/volumes/kubernetes.io~projected/kube-api-access-89mmd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b722d0a-df66-409f-a7cf-0a891f5a1d32/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0b722d0a-df66-409f-a7cf-0a891f5a1d32/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec/volumes/kubernetes.io~projected/kube-api-access-t5pnq:{mountpoint:/var/lib/kubelet/pods/1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec/volumes/kubernetes.io~projected/kube-api-access-t5pnq major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1eb02066-c20a-45b3-885a-0e9834b1c998/volumes/kubernetes.io~projected/kube-api-access-xhlrf:{mountpoint:/var/lib/kubelet/pods/1eb02066-c20a-45b3-885a-0e9834b1c998/volumes/kubernetes.io~projected/kube-api-access-xhlrf major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2104714e-6924-4133-90b9-7d98135901e8/volumes/kubernetes.io~projected/kube-api-access-5jzl4:{mountpoint:/var/lib/kubelet/pods/2104714e-6924-4133-90b9-7d98135901e8/volumes/kubernetes.io~projected/kube-api-access-5jzl4 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd/volumes/kubernetes.io~projected/kube-api-access-cwsq8:{mountpoint:/var/lib/kubelet/pods/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd/volumes/kubernetes.io~projected/kube-api-access-cwsq8 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b099592-879f-4313-be96-60d39b44f8a8/volumes/kubernetes.io~projected/kube-api-access-nr54g:{mountpoint:/var/lib/kubelet/pods/2b099592-879f-4313-be96-60d39b44f8a8/volumes/kubernetes.io~projected/kube-api-access-nr54g major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b099592-879f-4313-be96-60d39b44f8a8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2b099592-879f-4313-be96-60d39b44f8a8/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f174941-0801-466a-8cef-c2da4faa916f/volumes/kubernetes.io~projected/kube-api-access-cqbfm:{mountpoint:/var/lib/kubelet/pods/2f174941-0801-466a-8cef-c2da4faa916f/volumes/kubernetes.io~projected/kube-api-access-cqbfm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/315726a9-3009-40f6-bf65-6818acd95046/volumes/kubernetes.io~projected/kube-api-access-vmvr6:{mountpoint:/var/lib/kubelet/pods/315726a9-3009-40f6-bf65-6818acd95046/volumes/kubernetes.io~projected/kube-api-access-vmvr6 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/315726a9-3009-40f6-bf65-6818acd95046/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/315726a9-3009-40f6-bf65-6818acd95046/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36beafc8-4aee-448b-8dfd-64f89637aa9f/volumes/kubernetes.io~projected/kube-api-access-cjm42:{mountpoint:/var/lib/kubelet/pods/36beafc8-4aee-448b-8dfd-64f89637aa9f/volumes/kubernetes.io~projected/kube-api-access-cjm42 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42da3e8e-febb-4e95-ae4c-c750c4bd3481/volumes/kubernetes.io~projected/kube-api-access-wktj9:{mountpoint:/var/lib/kubelet/pods/42da3e8e-febb-4e95-ae4c-c750c4bd3481/volumes/kubernetes.io~projected/kube-api-access-wktj9 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f/volumes/kubernetes.io~projected/kube-api-access-k4tsh:{mountpoint:/var/lib/kubelet/pods/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f/volumes/kubernetes.io~projected/kube-api-access-k4tsh major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f/volumes/kubernetes.io~secret/serving-cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56b77a5b-a55e-4849-9789-b9c5a3db4b02/volumes/kubernetes.io~projected/kube-api-access-dqgfp:{mountpoint:/var/lib/kubelet/pods/56b77a5b-a55e-4849-9789-b9c5a3db4b02/volumes/kubernetes.io~projected/kube-api-access-dqgfp major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5732cb6f-524c-4389-8fe2-f81885322ae5/volumes/kubernetes.io~projected/kube-api-access-2mwqc:{mountpoint:/var/lib/kubelet/pods/5732cb6f-524c-4389-8fe2-f81885322ae5/volumes/kubernetes.io~projected/kube-api-access-2mwqc major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58637e90-c0b9-44da-91e9-7e1075a2636e/volumes/kubernetes.io~projected/kube-api-access-gsz5c:{mountpoint:/var/lib/kubelet/pods/58637e90-c0b9-44da-91e9-7e1075a2636e/volumes/kubernetes.io~projected/kube-api-access-gsz5c major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~projected/kube-api-access-79lwr:{mountpoint:/var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~projected/kube-api-access-79lwr major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7502a388-bbcd-4916-8b8d-a4258c58d575/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7502a388-bbcd-4916-8b8d-a4258c58d575/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76c09b47-0ab3-4212-924b-131d88e94ab3/volumes/kubernetes.io~projected/kube-api-access-q8z96:{mountpoint:/var/lib/kubelet/pods/76c09b47-0ab3-4212-924b-131d88e94ab3/volumes/kubernetes.io~projected/kube-api-access-q8z96 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76c09b47-0ab3-4212-924b-131d88e94ab3/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/76c09b47-0ab3-4212-924b-131d88e94ab3/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76c09b47-0ab3-4212-924b-131d88e94ab3/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/76c09b47-0ab3-4212-924b-131d88e94ab3/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c48a6b-a95c-4c64-9d89-1ed9d0320908/volumes/kubernetes.io~projected/kube-api-access-g5g9x:{mountpoint:/var/lib/kubelet/pods/80c48a6b-a95c-4c64-9d89-1ed9d0320908/volumes/kubernetes.io~projected/kube-api-access-g5g9x major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c48a6b-a95c-4c64-9d89-1ed9d0320908/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/80c48a6b-a95c-4c64-9d89-1ed9d0320908/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bfd499-9f29-4d6e-b2de-094d011b4f0b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/85bfd499-9f29-4d6e-b2de-094d011b4f0b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bfd499-9f29-4d6e-b2de-094d011b4f0b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/85bfd499-9f29-4d6e-b2de-094d011b4f0b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bfd499-9f29-4d6e-b2de-094d011b4f0b/volumes/kubernetes.io~projected/kube-api-access-jxwvv:{mountpoint:/var/lib/kubelet/pods/85bfd499-9f29-4d6e-b2de-094d011b4f0b/volumes/kubernetes.io~projected/kube-api-access-jxwvv major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/893a8c11-60f6-4736-9971-10520459a1d0/volumes/kubernetes.io~projected/kube-api-access-67xpb:{mountpoint:/var/lib/kubelet/pods/893a8c11-60f6-4736-9971-10520459a1d0/volumes/kubernetes.io~projected/kube-api-access-67xpb major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99c39ff9-f070-4c29-8284-4f98d116faa4/volumes/kubernetes.io~projected/kube-api-access-lkw2l:{mountpoint:/var/lib/kubelet/pods/99c39ff9-f070-4c29-8284-4f98d116faa4/volumes/kubernetes.io~projected/kube-api-access-lkw2l major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ef729d9-2e44-4c39-a018-5df7a941dcc5/volumes/kubernetes.io~projected/kube-api-access-4znfx:{mountpoint:/var/lib/kubelet/pods/9ef729d9-2e44-4c39-a018-5df7a941dcc5/volumes/kubernetes.io~projected/kube-api-access-4znfx major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd3a09f2-9d41-40f6-94f1-6cb00971d950/volumes/kubernetes.io~projected/kube-api-access-864s8:{mountpoint:/var/lib/kubelet/pods/bd3a09f2-9d41-40f6-94f1-6cb00971d950/volumes/kubernetes.io~projected/kube-api-access-864s8 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73ab4e1-b560-420b-8216-6272bd184f39/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d73ab4e1-b560-420b-8216-6272bd184f39/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db236f0a-264d-4498-8dd1-1787e2a3ee95/volumes/kubernetes.io~projected/kube-api-access-5fm7p:{mountpoint:/var/lib/kubelet/pods/db236f0a-264d-4498-8dd1-1787e2a3ee95/volumes/kubernetes.io~projected/kube-api-access-5fm7p major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~projected/kube-api-access-gl4ch:{mountpoint:/var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~projected/kube-api-access-gl4ch major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ebddd32199ee4b5ca18f8b880cc0a52ecffe1abfbb22e28c32abde22457512aa/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/877c1a0e4628300e7dbdbe1500c191cdb05b766c19c4e8807ed550ac73cbfcf2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b3ba6be7f92e9a5962830c37a1943080d7beafe79c8348e543d23b9691c45978/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/513ada148267a2b5b894fadbb38dd28747d957f25bb50298a62a3b5bbee694cc/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6413cae5397a7f8b4b4f08cd2cd24d1b59879ba0eb006b1d820e6ab4bb1282de/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9c515a869e25aa0f6c8ebad16c60680e284669a873926a74c03a6b59c3b3b990/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/88d53862bb3374932d604c9de9969ce538b79efcd71d5ddb437a09bcf63b08b6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/78c74e68e0dd355077f93b954732e2bf098685ae6a457f0f474e3a2d84109e0f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/bc27834b8715ac0c77ebce7393db5691dd1616930229419b220dead4df40f05d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/efe9402b9f702f1d89760b0e34b5abd4e55dcad367eb563f6d10b2e2cf8775b1/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/963c3bd0b976caf03e3ea5ddc7022dd1108d6e788ef947c3194e1e561afba51a/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/8f252d026d528ea10e5b38111377f2c7cd36c8c97e371f5d466f4fa77fa37722/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/81da34ab14b6743bb38c9afb9eacf97e27f031b24db7193907d02766edd88158/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c7a8edc8272409e6e57c5bc09303676ac28b4693f37b74c0916de129487079a2/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/1672a5547798ae94aea725da0c171b2d9e1b1c4b13d7a884ef956da1141012ef/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/108f64fe33a9672904c5a2bd1067cdbb18aca09b7a06a594805ba38097092c82/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/53ba22cce5d8dbfb639d3a874fa2830f04251ab5e4d57868e012b8bbd0898d4c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/26c17300fc907cc5bdc9b7c3e559836bf5f1af38af79799cfbf19af8ba5e539c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/363981afe73b52fb3606c61f4f1b142e296e98c1b7c5d58f7abbae62b9229b91/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0ac9a8d2fe71b8f4e98d03a627edea6c5bc932c505630c93b7237bda4539af2d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ddd39537788b3bb4af093e78bbf33aa083a2b1874269eef3e8f0932f74a4df3b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/7c074e7af0813adbb5e4f8482b3d90dc681ede117b075d320babb10498e635da/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/f808c2ac0d14d6e0ad80c80cdb38aa573d21bb9cc0c47659672960060fa4f431/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/6f8bdf8868bf43adce9a52d6c1e58ddf5ff34b914ddb51a43d310845c79ce324/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/35491cd3ff5d95903c9bf6949cbcbef22c43211fe5b36ca90388f6de8036a3a2/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/93050bcd9346a4ac0d08ea954ca41ae113f538fdae1246cf55d4f1d84e938445/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/975571331cbf6d2acc06c5027fce3b744c13e0b9cd944fbac21de4ecc91d90e1/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/037642006ed40e34ffc4104edd0f074dc439ade569387998078f6929f148b4a1/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/0cd0ecd29360d7ecd026f0a66c23ca202194fe7fa4eb06983ce2134b1a91f0f5/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/40a9bbadb43a08bfb1009a66948762fc3b9d3ba6d601bd258494e56df444a7e3/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/02a7da0b4bcec592446c4de72d813c997b5c38bd9f96a0b057f64f36487bf0e3/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/8bb3d869de256d842bb23792e36dac2a8c40e0ceee461e3b418512ef5ebd8a8d/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/c4cfb8389ee9d6970f80c7e865f5356453e9e8c32eaa46910521f4edcd1d03a4/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/3b8d767949037406bd73207b4ce4512d2a3b826ea4fd082fa4b2f6979c8e30d5/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/bf07fea6224cd58773b1fe8fc7fb2d7bacc3e6912d3dd0d0bc8b978ec27d50c2/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/db762d9e7520024df84bb2a3fa3e1af2dda36ad3926d54092cf9ae1aaed0fd2c/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/e99564d80263b02cb1cd01730d693b91d39a240917ba42e0a07f088ab2a019f2/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/7f7d336ab2687f0a7fe953c06e9cff200c70d137f3e344c287318b19422ba989/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/facdbffd427482b2e8d86cb06b01e83852241adaade43a2bd2a15ad35be7ebf3/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/d1ecf97cb8f983b0cf70c82d26eb1ce0e5945a1e4f8b994178c63ae1fec32e7d/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/0316c282f6bb8e985bd21ff84bdc7e7ee617cf6f2b018307f491b0eb4f6a7080/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/5aa0a6f0547daaa4cafb50c5fd7957a63c88f7f7e8bb0202ee53d6a5d7f5e795/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/effb2697152beabc2e7dac4d86edea8b06f8b15725a6fe4f276de85f67772844/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/93b023ba5b9b09dc8abee942fcda962a9adf4cb5d76f86b0f033368a7ef09c06/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/985b337a4afe130f45a574674eb89b217dfa1061836d60c09535fdbfbf6b6e92/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f8537993d6378115aca0b31b1179e4b21c97a1146bbef0e2c9537b5c0fe3b811/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/af482657de7da71dacee8373839d43533995e6347825ac104b20dbb9d22ae87e/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/7dfd803d3237344a5ad17b228db1a427e00313173dfbd2ea6ac81585ecdbe62f/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/4e6f2d81b4309c78af9082b41a57e81c48e3e530a6d4c6a9916671669a3e11d4/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/bd8ee8ed94bc5eef3d24b0987fdcd626a95d3d83e50889532541948646295ca8/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ee01b358dffd02ecc2024993397323d5cd30f9a109ca313d23626a37344bf143/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1e63889b99314aea6b66f7ccad4b96e7fb202d68333534cc46fa95db0bafaa66/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c0f38c7c7927778d8e3eebba6ee9b60749358e3d3c72bea05b6c8ca1dbace0a4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f454605d7db119e393068b84a69c05f999d9136fce815710b9c44e11c4027323/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.280377 4855 manager.go:217] Machine: {Timestamp:2026-11-03 05:55:12.279914383 +0000 UTC m=+0.088301643 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099880 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec253fb47201fedfc553bc4d6b160c53 SystemUUID:ec253fb4-7201-fedf-c553-bc4d6b160c53 BootID:45f85e64-bb2b-460c-becd-652a8ffa59c1 Filesystems:[{Device:/var/lib/kubelet/pods/1eb02066-c20a-45b3-885a-0e9834b1c998/volumes/kubernetes.io~projected/kube-api-access-xhlrf DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7e9d06a2a1bd320cf8e4d6451881ffe3de535e8906f94c326b01ca451574213/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80c48a6b-a95c-4c64-9d89-1ed9d0320908/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~projected/kube-api-access-79lwr DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b722d0a-df66-409f-a7cf-0a891f5a1d32/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80c48a6b-a95c-4c64-9d89-1ed9d0320908/volumes/kubernetes.io~projected/kube-api-access-g5g9x DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1fcf9b67df930930c3aa224ef49c55b3fd82d8b92b15764d2fb6e7a0890cfff/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee9b430004788822fe9777f5b33e99c1c93ea7f922c7ddac81d34c7980535505/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99c39ff9-f070-4c29-8284-4f98d116faa4/volumes/kubernetes.io~projected/kube-api-access-lkw2l DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/2f174941-0801-466a-8cef-c2da4faa916f/volumes/kubernetes.io~projected/kube-api-access-cqbfm DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5f9a19056772b54b900cbd373eca0b0ce02f0484ad370a914463ec0753e00dd/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/315726a9-3009-40f6-bf65-6818acd95046/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2104714e-6924-4133-90b9-7d98135901e8/volumes/kubernetes.io~projected/kube-api-access-5jzl4 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/059a8188-2f82-4b0d-af82-5598a6e7b50e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/315726a9-3009-40f6-bf65-6818acd95046/volumes/kubernetes.io~projected/kube-api-access-vmvr6 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5732cb6f-524c-4389-8fe2-f81885322ae5/volumes/kubernetes.io~projected/kube-api-access-2mwqc DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b099592-879f-4313-be96-60d39b44f8a8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b722d0a-df66-409f-a7cf-0a891f5a1d32/volumes/kubernetes.io~projected/kube-api-access-89mmd DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76c09b47-0ab3-4212-924b-131d88e94ab3/volumes/kubernetes.io~projected/kube-api-access-q8z96 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd/volumes/kubernetes.io~projected/kube-api-access-cwsq8 DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/85bfd499-9f29-4d6e-b2de-094d011b4f0b/volumes/kubernetes.io~projected/kube-api-access-jxwvv DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6dd0b7a3516914b9370f372e6e54c49b972ea78d5a7ccc80fa468db31d7edd9d/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd3a09f2-9d41-40f6-94f1-6cb00971d950/volumes/kubernetes.io~projected/kube-api-access-864s8 DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/893a8c11-60f6-4736-9971-10520459a1d0/volumes/kubernetes.io~projected/kube-api-access-67xpb DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42da3e8e-febb-4e95-ae4c-c750c4bd3481/volumes/kubernetes.io~projected/kube-api-access-wktj9 DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/189c597ef65ce3c5390ace18f63184a962739e8fdb474b724c16c94e528534d8/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1081d6240a8fba18caf5647b6ada31caccf4aeb3a962329ccd5675689030b54e/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e32bf93d77ff76fa461e71ab334be2d22802595e294984e161f19b8ab63e28e5/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~projected/kube-api-access-gl4ch DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ea405697ad2752b6132fc12e652c621b186fac158e520481546aa51e18a8d70/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08a92738-2cd0-4eed-9e83-c01baed110c2/volumes/kubernetes.io~projected/kube-api-access-k7hfp DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf72b91612df2fa8a4c1f58f81062a6ea5c9c64f16e05ac77b5bd370024b970f/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/85bfd499-9f29-4d6e-b2de-094d011b4f0b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f/volumes/kubernetes.io~projected/kube-api-access-k4tsh DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76c09b47-0ab3-4212-924b-131d88e94ab3/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5624d691f94f024b3a87e1b0882e0b03e3ee89535ab7e8ac3a47a52ebf962e2e/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/054b9d40-fa74-4f73-9db4-70ff0dd43191/volumes/kubernetes.io~projected/kube-api-access-zp2g4 DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58637e90-c0b9-44da-91e9-7e1075a2636e/volumes/kubernetes.io~projected/kube-api-access-gsz5c DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/718c715c7de938642fa7a115a4af073f8ac12f8aba6373708322979065e4f003/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db236f0a-264d-4498-8dd1-1787e2a3ee95/volumes/kubernetes.io~projected/kube-api-access-5fm7p DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24334de0cccdb5ad2f73dd53e8955dccdfeb342780e7de20d73aaf49c830c0fa/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85bfd499-9f29-4d6e-b2de-094d011b4f0b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7502a388-bbcd-4916-8b8d-a4258c58d575/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ec17c2466431329949305a0981ed793552f6226ff216836e2d1be2e6e904ce1/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56b77a5b-a55e-4849-9789-b9c5a3db4b02/volumes/kubernetes.io~projected/kube-api-access-dqgfp DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76c09b47-0ab3-4212-924b-131d88e94ab3/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e9a399c0d95841d6e79f3805b1ce884a2f866ce675185df36cc0cf6d62737b6/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0f1d323aff656dfbaf8d35db010d71881b756f47b15ea263f2040ca0188a2fb/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ef729d9-2e44-4c39-a018-5df7a941dcc5/volumes/kubernetes.io~projected/kube-api-access-4znfx DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b099592-879f-4313-be96-60d39b44f8a8/volumes/kubernetes.io~projected/kube-api-access-nr54g DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/649c95861450533aae74aa2e6b4a98dd10706551a5f6c510e57802b6561ad0f5/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/683bb386ac9f4af2f1ac192455504ae55d1098799acc53eacbf3cd9d40c31469/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/059a8188-2f82-4b0d-af82-5598a6e7b50e/volumes/kubernetes.io~projected/kube-api-access-tf7vt DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d73ab4e1-b560-420b-8216-6272bd184f39/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36beafc8-4aee-448b-8dfd-64f89637aa9f/volumes/kubernetes.io~projected/kube-api-access-cjm42 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec/volumes/kubernetes.io~projected/kube-api-access-t5pnq DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc273797706cc875480faf0a6abdfb6868d5972e4e8d767eb4523c02b7f71edc/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/089f26e348dc1b9a0fa160f005b1cfafce0b63c24e1e9fa4597d30eace4648dc/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b549c16578dd87e9bdf0b6ac99e5583d0b7c04dd077ab74fcde8d5a2de7f045c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b722d0a-df66-409f-a7cf-0a891f5a1d32/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25e3df992bc69f8ac07b758425f8e2a70002aea8f351e9d65368ae1b2ad3fcc1/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1081d6240a8fba1 MacAddress:86:7c:91:13:74:fc Speed:10000 Mtu:8901} {Name:189c597ef65ce3c MacAddress:56:e8:73:26:e4:ae Speed:10000 Mtu:8901} {Name:24334de0cccdb5a MacAddress:92:3d:51:77:3f:75 Speed:10000 Mtu:8901} {Name:25e3df992bc69f8 MacAddress:da:86:f4:1c:5e:48 Speed:10000 Mtu:8901} {Name:5ea405697ad2752 MacAddress:56:73:72:89:69:07 Speed:10000 Mtu:8901} {Name:683bb386ac9f4af MacAddress:02:19:6e:73:bf:38 Speed:10000 Mtu:8901} {Name:6e9a399c0d95841 MacAddress:32:a4:ed:2c:70:2e Speed:10000 Mtu:8901} {Name:bf72b91612df2fa MacAddress:ea:f0:7b:c0:b5:37 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e1:fc:0f:69 Speed:0 Mtu:9001} {Name:br-int MacAddress:46:a8:05:09:02:be Speed:0 Mtu:8901} {Name:c0f1d323aff656d MacAddress:72:ae:0f:03:30:65 Speed:10000 Mtu:8901} {Name:cc273797706cc87 MacAddress:9e:dc:34:c0:7a:ae Speed:10000 Mtu:8901} {Name:e1fcf9b67df9309 MacAddress:96:a2:af:fe:3b:66 Speed:10000 Mtu:8901} {Name:e32bf93d77ff76f MacAddress:be:d8:d3:c0:8d:bc Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e1:fc:0f:69 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:fe:e9:48:86:84:01 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8e:08:38:50:35:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.280774 4855 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.280826 4855 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.281279 4855 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.281318 4855 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-67.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.281555 4855 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.281568 4855 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.281582 4855 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.281600 4855 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.281628 4855 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.281895 4855 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.282041 4855 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.282086 4855 kubelet.go:493] "Attempting to sync node with API server" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.282100 4855 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.282114 4855 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.282128 4855 kubelet.go:405] "Adding apiserver pod source" Nov 03 05:55:12.283253 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.282139 4855 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 05:55:12.284066 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.283320 4855 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 05:55:12.284066 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.283334 4855 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 05:55:12.284066 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.284035 4855 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 05:55:12.286315 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286300 4855 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 05:55:12.286315 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286315 4855 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 05:55:12.286436 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286425 4855 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 05:55:12.286623 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286612 4855 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 05:55:12.286662 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286642 4855 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 05:55:12.286774 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286760 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 05:55:12.286803 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286788 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 05:55:12.286834 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286802 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 05:55:12.286834 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286814 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 05:55:12.286834 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286826 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 05:55:12.286914 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286839 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 05:55:12.286914 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286867 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 05:55:12.286914 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286881 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 05:55:12.286914 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286895 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 05:55:12.286914 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286908 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 05:55:12.287048 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286934 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 05:55:12.287048 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286954 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 05:55:12.287048 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.286998 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 05:55:12.287048 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.287010 4855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 05:55:12.290532 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.290517 4855 server.go:1263] "Started kubelet" Nov 03 05:55:12.290732 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.290607 4855 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 05:55:12.290732 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.290632 4855 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 05:55:12.290732 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.290723 4855 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 05:55:12.291057 ip-10-0-141-67 systemd[1]: Started Kubernetes Kubelet. Nov 03 05:55:12.291251 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.291085 4855 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 05:55:12.295286 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.295270 4855 server.go:317] "Adding debug handlers to kubelet server" Nov 03 05:55:12.299059 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.296344 4855 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 05:55:12.299059 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.296367 4855 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 05:55:12.299059 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.296436 4855 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 05:49:34 +0000 UTC" deadline="2028-06-20 19:41:12.931878372 +0000 UTC" Nov 03 05:55:12.299059 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.296457 4855 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14293h46m0.635424251s" Nov 03 05:55:12.299059 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.296462 4855 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 05:55:12.299059 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.296472 4855 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 05:55:12.299059 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:12.296486 4855 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-67.ec2.internal\" not found" Nov 03 05:55:12.299059 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.296920 4855 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 05:55:12.299059 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.298981 4855 factory.go:55] Registering systemd factory Nov 03 05:55:12.299059 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.298997 4855 factory.go:223] Registration of the systemd container factory successfully Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.300919 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="059a8188-2f82-4b0d-af82-5598a6e7b50e" volumeName="kubernetes.io/projected/059a8188-2f82-4b0d-af82-5598a6e7b50e-kube-api-access-tf7vt" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.300958 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a0e103-b720-45e0-8f30-c54e9542b5cf" volumeName="kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-bound-sa-token" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.300972 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c48a6b-a95c-4c64-9d89-1ed9d0320908" volumeName="kubernetes.io/secret/80c48a6b-a95c-4c64-9d89-1ed9d0320908-serving-cert" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.300983 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba1e446-7be2-4673-9a44-a6c6a5e1be2d" volumeName="kubernetes.io/configmap/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-nginx-conf" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.300993 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3a09f2-9d41-40f6-94f1-6cb00971d950" volumeName="kubernetes.io/empty-dir/bd3a09f2-9d41-40f6-94f1-6cb00971d950-tmp-dir" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301005 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b722d0a-df66-409f-a7cf-0a891f5a1d32" volumeName="kubernetes.io/projected/0b722d0a-df66-409f-a7cf-0a891f5a1d32-kube-api-access-89mmd" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301017 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b099592-879f-4313-be96-60d39b44f8a8" volumeName="kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-trusted-ca" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301028 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f" volumeName="kubernetes.io/configmap/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-config" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301043 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a0e103-b720-45e0-8f30-c54e9542b5cf" volumeName="kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-certificates" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301054 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a0e103-b720-45e0-8f30-c54e9542b5cf" volumeName="kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-image-registry-private-configuration" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301065 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfd499-9f29-4d6e-b2de-094d011b4f0b" volumeName="kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-tuned" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301077 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b099592-879f-4313-be96-60d39b44f8a8" volumeName="kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-config" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301088 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="315726a9-3009-40f6-bf65-6818acd95046" volumeName="kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-trusted-ca-bundle" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301100 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a0e103-b720-45e0-8f30-c54e9542b5cf" volumeName="kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-installation-pull-secrets" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301114 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db236f0a-264d-4498-8dd1-1787e2a3ee95" volumeName="kubernetes.io/projected/db236f0a-264d-4498-8dd1-1787e2a3ee95-kube-api-access-5fm7p" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301125 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="059a8188-2f82-4b0d-af82-5598a6e7b50e" volumeName="kubernetes.io/secret/059a8188-2f82-4b0d-af82-5598a6e7b50e-cert" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301137 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08a92738-2cd0-4eed-9e83-c01baed110c2" volumeName="kubernetes.io/empty-dir/08a92738-2cd0-4eed-9e83-c01baed110c2-data-volume" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301149 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f" volumeName="kubernetes.io/secret/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-serving-cert" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301160 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08a92738-2cd0-4eed-9e83-c01baed110c2" volumeName="kubernetes.io/projected/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-api-access-k7hfp" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301172 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="219a457b-c7b8-4a1d-8f14-50a5d56bb9bd" volumeName="kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301187 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5732cb6f-524c-4389-8fe2-f81885322ae5" volumeName="kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301199 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58637e90-c0b9-44da-91e9-7e1075a2636e" volumeName="kubernetes.io/projected/58637e90-c0b9-44da-91e9-7e1075a2636e-kube-api-access-gsz5c" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301210 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" volumeName="kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-trusted-ca" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301221 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" volumeName="kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-kube-api-access-gl4ch" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301233 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2104714e-6924-4133-90b9-7d98135901e8" volumeName="kubernetes.io/projected/2104714e-6924-4133-90b9-7d98135901e8-kube-api-access-5jzl4" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301245 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="315726a9-3009-40f6-bf65-6818acd95046" volumeName="kubernetes.io/projected/315726a9-3009-40f6-bf65-6818acd95046-kube-api-access-vmvr6" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301264 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42da3e8e-febb-4e95-ae4c-c750c4bd3481" volumeName="kubernetes.io/projected/42da3e8e-febb-4e95-ae4c-c750c4bd3481-kube-api-access-wktj9" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301274 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" volumeName="kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-certificates" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301295 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b099592-879f-4313-be96-60d39b44f8a8" volumeName="kubernetes.io/projected/2b099592-879f-4313-be96-60d39b44f8a8-kube-api-access-nr54g" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301307 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a0e103-b720-45e0-8f30-c54e9542b5cf" volumeName="kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-trusted-ca" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301319 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76c09b47-0ab3-4212-924b-131d88e94ab3" volumeName="kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-stats-auth" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301331 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7502a388-bbcd-4916-8b8d-a4258c58d575" volumeName="kubernetes.io/configmap/7502a388-bbcd-4916-8b8d-a4258c58d575-konnectivity-ca" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301343 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b722d0a-df66-409f-a7cf-0a891f5a1d32" volumeName="kubernetes.io/secret/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301354 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb02066-c20a-45b3-885a-0e9834b1c998" volumeName="kubernetes.io/configmap/1eb02066-c20a-45b3-885a-0e9834b1c998-serviceca" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301366 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb02066-c20a-45b3-885a-0e9834b1c998" volumeName="kubernetes.io/projected/1eb02066-c20a-45b3-885a-0e9834b1c998-kube-api-access-xhlrf" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301377 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f174941-0801-466a-8cef-c2da4faa916f" volumeName="kubernetes.io/projected/2f174941-0801-466a-8cef-c2da4faa916f-kube-api-access-cqbfm" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301407 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="315726a9-3009-40f6-bf65-6818acd95046" volumeName="kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-snapshots" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301418 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5732cb6f-524c-4389-8fe2-f81885322ae5" volumeName="kubernetes.io/projected/5732cb6f-524c-4389-8fe2-f81885322ae5-kube-api-access-2mwqc" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301430 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3a09f2-9d41-40f6-94f1-6cb00971d950" volumeName="kubernetes.io/configmap/bd3a09f2-9d41-40f6-94f1-6cb00971d950-config-volume" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301441 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b722d0a-df66-409f-a7cf-0a891f5a1d32" volumeName="kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-config" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301452 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b722d0a-df66-409f-a7cf-0a891f5a1d32" volumeName="kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-script-lib" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301465 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2104714e-6924-4133-90b9-7d98135901e8" volumeName="kubernetes.io/configmap/2104714e-6924-4133-90b9-7d98135901e8-iptables-alerter-script" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301478 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="315726a9-3009-40f6-bf65-6818acd95046" volumeName="kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-service-ca-bundle" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301489 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f" volumeName="kubernetes.io/projected/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-kube-api-access-k4tsh" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301500 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56b77a5b-a55e-4849-9789-b9c5a3db4b02" volumeName="kubernetes.io/projected/56b77a5b-a55e-4849-9789-b9c5a3db4b02-kube-api-access-dqgfp" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301510 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c39ff9-f070-4c29-8284-4f98d116faa4" volumeName="kubernetes.io/configmap/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-config" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301522 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3a09f2-9d41-40f6-94f1-6cb00971d950" volumeName="kubernetes.io/projected/bd3a09f2-9d41-40f6-94f1-6cb00971d950-kube-api-access-864s8" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301534 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76c09b47-0ab3-4212-924b-131d88e94ab3" volumeName="kubernetes.io/projected/76c09b47-0ab3-4212-924b-131d88e94ab3-kube-api-access-q8z96" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301546 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c48a6b-a95c-4c64-9d89-1ed9d0320908" volumeName="kubernetes.io/projected/80c48a6b-a95c-4c64-9d89-1ed9d0320908-kube-api-access-g5g9x" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301557 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="315726a9-3009-40f6-bf65-6818acd95046" volumeName="kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-tmp" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301568 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a0e103-b720-45e0-8f30-c54e9542b5cf" volumeName="kubernetes.io/empty-dir/74a0e103-b720-45e0-8f30-c54e9542b5cf-ca-trust-extracted" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301580 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c39ff9-f070-4c29-8284-4f98d116faa4" volumeName="kubernetes.io/projected/99c39ff9-f070-4c29-8284-4f98d116faa4-kube-api-access-lkw2l" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301591 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" volumeName="kubernetes.io/empty-dir/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-ca-trust-extracted" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301604 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" volumeName="kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-installation-pull-secrets" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301615 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b722d0a-df66-409f-a7cf-0a891f5a1d32" volumeName="kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-env-overrides" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301625 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f174941-0801-466a-8cef-c2da4faa916f" volumeName="kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-cni-binary-copy" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301646 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="315726a9-3009-40f6-bf65-6818acd95046" volumeName="kubernetes.io/secret/315726a9-3009-40f6-bf65-6818acd95046-serving-cert" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301656 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5732cb6f-524c-4389-8fe2-f81885322ae5" volumeName="kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301667 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73ab4e1-b560-420b-8216-6272bd184f39" volumeName="kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301680 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" volumeName="kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-image-registry-private-configuration" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301692 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5732cb6f-524c-4389-8fe2-f81885322ae5" volumeName="kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-binary-copy" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301703 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76c09b47-0ab3-4212-924b-131d88e94ab3" volumeName="kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-default-certificate" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301715 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56b77a5b-a55e-4849-9789-b9c5a3db4b02" volumeName="kubernetes.io/configmap/56b77a5b-a55e-4849-9789-b9c5a3db4b02-telemetry-config" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301726 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c48a6b-a95c-4c64-9d89-1ed9d0320908" volumeName="kubernetes.io/configmap/80c48a6b-a95c-4c64-9d89-1ed9d0320908-config" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301737 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36beafc8-4aee-448b-8dfd-64f89637aa9f" volumeName="kubernetes.io/projected/36beafc8-4aee-448b-8dfd-64f89637aa9f-kube-api-access-cjm42" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301749 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef729d9-2e44-4c39-a018-5df7a941dcc5" volumeName="kubernetes.io/projected/9ef729d9-2e44-4c39-a018-5df7a941dcc5-kube-api-access-4znfx" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301760 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" volumeName="kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-bound-sa-token" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301771 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b099592-879f-4313-be96-60d39b44f8a8" volumeName="kubernetes.io/secret/2b099592-879f-4313-be96-60d39b44f8a8-serving-cert" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301783 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36beafc8-4aee-448b-8dfd-64f89637aa9f" volumeName="kubernetes.io/empty-dir/36beafc8-4aee-448b-8dfd-64f89637aa9f-tmp-dir" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301794 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfd499-9f29-4d6e-b2de-094d011b4f0b" volumeName="kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-tmp" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301805 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfd499-9f29-4d6e-b2de-094d011b4f0b" volumeName="kubernetes.io/projected/85bfd499-9f29-4d6e-b2de-094d011b4f0b-kube-api-access-jxwvv" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301817 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="893a8c11-60f6-4736-9971-10520459a1d0" volumeName="kubernetes.io/projected/893a8c11-60f6-4736-9971-10520459a1d0-kube-api-access-67xpb" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301830 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec" volumeName="kubernetes.io/projected/1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec-kube-api-access-t5pnq" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301842 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f174941-0801-466a-8cef-c2da4faa916f" volumeName="kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-multus-daemon-config" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301854 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a0e103-b720-45e0-8f30-c54e9542b5cf" volumeName="kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-kube-api-access-79lwr" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301866 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7502a388-bbcd-4916-8b8d-a4258c58d575" volumeName="kubernetes.io/secret/7502a388-bbcd-4916-8b8d-a4258c58d575-agent-certs" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301877 4855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="054b9d40-fa74-4f73-9db4-70ff0dd43191" volumeName="kubernetes.io/projected/054b9d40-fa74-4f73-9db4-70ff0dd43191-kube-api-access-zp2g4" seLinuxMountContext="" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301888 4855 reconstruct.go:97] "Volume reconstruction finished" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.301896 4855 reconciler.go:29] "Reconciler: start to sync state" Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.302316 4855 factory.go:153] Registering CRI-O factory Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.302335 4855 factory.go:223] Registration of the crio container factory successfully Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.302408 4855 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.302432 4855 factory.go:103] Registering Raw factory Nov 03 05:55:12.302410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.302450 4855 manager.go:1199] Started watching for new ooms in manager Nov 03 05:55:12.306931 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.302802 4855 manager.go:319] Starting recovery of all containers Nov 03 05:55:12.319676 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.319508 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/1.log" Nov 03 05:55:12.319676 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.319633 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/1.log" Nov 03 05:55:12.323957 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.323840 4855 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 05:55:12.325474 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.325450 4855 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 05:55:12.325474 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.325474 4855 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 05:55:12.325657 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.325495 4855 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 05:55:12.325657 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:12.325539 4855 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 05:55:12.336312 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.336278 4855 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 05:55:12.350369 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.350289 4855 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 05:55:12.350702 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.350666 4855 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="095019cc6d00ad522b05a58baab581877979711bfad98f14b2ef9e82949e6435" exitCode=0 Nov 03 05:55:12.350702 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.350702 4855 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="8e57a6ba76f01c8db94bc7b21b40a639c5b33033198345e331427bb02de6fa95" exitCode=0 Nov 03 05:55:12.350834 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.350715 4855 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="195d0cedb16bbd0aa18b28e8af1ee4a18033cc13583f6e6c14dfca3ca0ab22f2" exitCode=0 Nov 03 05:55:12.350834 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.350728 4855 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="4f915b2edd5f43c4f26437b9298c05548fadac00bcf366a7feb915adf334da18" exitCode=0 Nov 03 05:55:12.350834 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.350738 4855 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="f6cc04c515aaaf642c4b79f84099c72d16af04656eecddc6b64cdf746da308ae" exitCode=0 Nov 03 05:55:12.350834 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.350747 4855 generic.go:356] "Generic (PLEG): container finished" podID="5732cb6f-524c-4389-8fe2-f81885322ae5" containerID="2331d1819ab747057808128e9cb00b4554cac81b2a29a6a548e0f970256ed234" exitCode=0 Nov 03 05:55:12.358790 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.358767 4855 generic.go:356] "Generic (PLEG): container finished" podID="18b012b3f595caea8c478eb61ba981af" containerID="7fae73732c0d329a9240c03a2c52dec2e97cc13e547075d7b4c5a95dc31cdf32" exitCode=0 Nov 03 05:55:12.363381 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.363353 4855 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 05:55:12.364214 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.364191 4855 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 05:55:12.364407 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.364377 4855 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 05:55:12.369380 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.369359 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/1.log" Nov 03 05:55:12.369477 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.369415 4855 generic.go:356] "Generic (PLEG): container finished" podID="2b099592-879f-4313-be96-60d39b44f8a8" containerID="a4debb86934e6ea1da15236ab15df42b19e3ca5cef0906b0ce4d4abfd1f63885" exitCode=255 Nov 03 05:55:12.389227 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.389210 4855 manager.go:324] Recovery completed Nov 03 05:55:12.409287 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409264 4855 cpu_manager.go:226] "Starting" policy="none" Nov 03 05:55:12.409287 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409278 4855 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 05:55:12.409426 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409296 4855 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 05:55:12.409460 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409429 4855 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 05:55:12.409460 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409440 4855 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 05:55:12.409460 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409455 4855 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 05:55:12.409460 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409460 4855 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 05:55:12.409589 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409466 4855 policy_none.go:50] "Start" Nov 03 05:55:12.409589 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409487 4855 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 05:55:12.409589 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409496 4855 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 05:55:12.409670 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409622 4855 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 05:55:12.409670 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409629 4855 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 05:55:12.409670 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.409634 4855 policy_none.go:44] "Start" Nov 03 05:55:12.413862 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.413835 4855 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 05:55:12.416536 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.416524 4855 manager.go:341] "Starting Device Plugin manager" Nov 03 05:55:12.416990 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:12.416548 4855 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 05:55:12.416990 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.416991 4855 server.go:87] "Starting device plugin registration server" Nov 03 05:55:12.417163 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.417150 4855 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 05:55:12.417209 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.417162 4855 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 05:55:12.417300 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.417277 4855 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 05:55:12.417541 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.417437 4855 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 05:55:12.417541 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.417479 4855 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 05:55:12.417541 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.417487 4855 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 05:55:12.417727 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.417518 4855 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T05:55:12.417467389Z","UUID":"bb44f4cd-c77c-49be-a508-a10ab2700955","Handler":null,"Name":"","Endpoint":""} Nov 03 05:55:12.419496 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.419464 4855 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 05:55:12.419589 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.419513 4855 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 05:55:12.420274 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:12.420153 4855 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 05:55:12.425644 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.425610 4855 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal"] Nov 03 05:55:12.426329 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.426311 4855 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.426483 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.426442 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" event={"ID":"18b012b3f595caea8c478eb61ba981af","Type":"ContainerStarted","Data":"5a38bc902652725cd42096bba832db2ed4c0ade10e99fd71c9dc38b8e4373bbd"} Nov 03 05:55:12.426572 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.426500 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" event={"ID":"18b012b3f595caea8c478eb61ba981af","Type":"ContainerDied","Data":"7fae73732c0d329a9240c03a2c52dec2e97cc13e547075d7b4c5a95dc31cdf32"} Nov 03 05:55:12.426572 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.426520 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" event={"ID":"18b012b3f595caea8c478eb61ba981af","Type":"ContainerStarted","Data":"ee9b430004788822fe9777f5b33e99c1c93ea7f922c7ddac81d34c7980535505"} Nov 03 05:55:12.426572 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.426536 4855 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.426572 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.426565 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" event={"ID":"45ef46b1ad95006153c299e1a6c8cea2","Type":"ContainerStarted","Data":"a7d91914558c7941ee95e7ad87dfe375271878ac1781b9d6d055b1b286965476"} Nov 03 05:55:12.426780 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.426582 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" event={"ID":"45ef46b1ad95006153c299e1a6c8cea2","Type":"ContainerStarted","Data":"649c95861450533aae74aa2e6b4a98dd10706551a5f6c510e57802b6561ad0f5"} Nov 03 05:55:12.451314 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.451294 4855 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 05:55:12.451414 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:12.451339 4855 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-67.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.451928 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.451912 4855 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 05:55:12.451982 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:12.451945 4855 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.517983 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.517958 4855 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 05:55:12.521401 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.521369 4855 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientMemory" Nov 03 05:55:12.521502 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.521422 4855 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 05:55:12.521502 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.521433 4855 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-67.ec2.internal" event="NodeHasSufficientPID" Nov 03 05:55:12.521502 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.521475 4855 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.551020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.550996 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.551116 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.551026 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45ef46b1ad95006153c299e1a6c8cea2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-67.ec2.internal\" (UID: \"45ef46b1ad95006153c299e1a6c8cea2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.551116 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.551043 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.558593 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.558576 4855 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.558637 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.558630 4855 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.651242 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.651157 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.651242 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.651196 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.651242 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.651224 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45ef46b1ad95006153c299e1a6c8cea2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-67.ec2.internal\" (UID: \"45ef46b1ad95006153c299e1a6c8cea2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.651474 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.651268 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.651474 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.651272 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45ef46b1ad95006153c299e1a6c8cea2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-67.ec2.internal\" (UID: \"45ef46b1ad95006153c299e1a6c8cea2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:55:12.651474 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:12.651274 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18b012b3f595caea8c478eb61ba981af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" (UID: \"18b012b3f595caea8c478eb61ba981af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:55:13.283197 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.283157 4855 apiserver.go:52] "Watching apiserver" Nov 03 05:55:13.347636 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.347612 4855 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 05:55:13.350013 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.349954 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9c7rn","openshift-dns/node-resolver-99n4t","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7","openshift-network-diagnostics/network-check-target-cp5xp","openshift-image-registry/image-registry-67898f7b5f-gmmrp","openshift-image-registry/node-ca-vvkcc","openshift-ingress/router-default-75b97bb597-swr58","openshift-insights/insights-runtime-extractor-tqt2f","kube-system/konnectivity-agent-sp2qh","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j","openshift-console-operator/console-operator-7cd56b8dbc-qwvxg","openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn","kube-system/kas-connection-checker-685895c7c9-bhkn5","kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk","openshift-image-registry/image-registry-7c96d9b8b6-5hh78","kube-system/kas-connection-checker-685895c7c9-tnzjq","openshift-network-console/networking-console-plugin-86c478bb88-zknxx","kube-system/global-pull-secret-syncer-vspq2","kube-system/kas-connection-checker-685895c7c9-6z5gg","openshift-multus/network-metrics-daemon-hbdpd","openshift-dns/dns-default-g9htv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal","openshift-network-operator/iptables-alerter-xzjql","openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq","openshift-cluster-node-tuning-operator/tuned-kkvkc","openshift-ingress-canary/ingress-canary-6j7pt","openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf","openshift-multus/multus-w4bpx","openshift-insights/insights-operator-7c4986d449-86pnb","openshift-ovn-kubernetes/ovnkube-node-c75h9"] Nov 03 05:55:13.351853 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.351824 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.352523 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.352500 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:13.353316 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.353295 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:13.354180 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.354162 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.354953 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.354937 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:13.358144 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.358123 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:13.361681 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.361663 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.362870 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.362856 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:13.363340 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.363324 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.369574 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.369558 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 05:55:13.370715 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.370696 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.370784 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.370734 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 05:55:13.370784 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.370740 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 05:55:13.370940 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.370925 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 05:55:13.371463 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.371451 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 05:55:13.371651 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.371641 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 05:55:13.371984 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.371969 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.372034 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.371983 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 05:55:13.372034 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.371987 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.372276 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.372260 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 05:55:13.372451 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.372440 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 05:55:13.372750 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.372738 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.372810 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.372789 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.374518 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.374505 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.375672 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.375641 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 05:55:13.376312 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.376290 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 05:55:13.376413 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.376360 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 05:55:13.376518 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.376424 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.376587 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.376571 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.376636 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.376623 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.376739 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.376680 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.376871 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.376854 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.376920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.376909 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.377013 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.377000 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.377057 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.377024 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.380199 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.380179 4855 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:55:13.380326 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.380312 4855 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:55:13.385466 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.385452 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 05:55:13.386285 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.386271 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.386524 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.386505 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xj6m8\"" Nov 03 05:55:13.386607 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.386519 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8hrbq\"" Nov 03 05:55:13.386726 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.386713 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zq86f\"" Nov 03 05:55:13.386783 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.386744 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 05:55:13.386783 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.386758 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 05:55:13.386841 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.386781 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.387050 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.387036 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.387050 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.387036 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.387140 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.387066 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nphqj\"" Nov 03 05:55:13.387169 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.387143 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 05:55:13.387276 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.387263 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.387325 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.387302 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 05:55:13.387325 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.387313 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.388443 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.388414 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 05:55:13.390441 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.390424 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 05:55:13.390513 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.390428 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 05:55:13.390513 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.390486 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 05:55:13.390837 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.390822 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z796t\"" Nov 03 05:55:13.390965 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.390949 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 05:55:13.391026 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.390971 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rwdxr\"" Nov 03 05:55:13.391469 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.391453 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 05:55:13.399291 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.399277 4855 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 05:55:13.399567 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.399552 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 05:55:13.399567 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.399563 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 05:55:13.399669 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.399570 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 05:55:13.399669 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.399595 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 05:55:13.400655 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400639 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fkxng\"" Nov 03 05:55:13.400720 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400663 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-krwzb\"" Nov 03 05:55:13.400720 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400681 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gfnmx\"" Nov 03 05:55:13.400720 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400703 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nvbtr\"" Nov 03 05:55:13.400720 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400712 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-97pnm\"" Nov 03 05:55:13.400879 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400717 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mlps4\"" Nov 03 05:55:13.400879 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400742 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xvf66\"" Nov 03 05:55:13.400879 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400749 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j9vz5\"" Nov 03 05:55:13.400879 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400761 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-75slc\"" Nov 03 05:55:13.400879 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400772 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.400879 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400794 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwk29\"" Nov 03 05:55:13.401057 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400882 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.401057 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.400905 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.401139 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401130 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.401188 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401160 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 05:55:13.401188 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401171 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.401188 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401180 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.401188 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401173 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.401330 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401184 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 05:55:13.401330 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401197 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 05:55:13.401330 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401205 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 05:55:13.401330 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401217 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 05:55:13.401498 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401482 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 05:55:13.401561 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401502 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 05:55:13.401561 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401512 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.401561 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401523 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.401561 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401540 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 05:55:13.401561 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401546 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.401561 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401561 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.401745 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401626 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 05:55:13.401745 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401631 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 05:55:13.401745 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401629 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 05:55:13.401984 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401971 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 05:55:13.402033 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.401989 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.402081 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.402040 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.402081 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.402050 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 05:55:13.402081 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.402055 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 05:55:13.402081 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.402050 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 05:55:13.402236 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.402130 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.402236 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.402161 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 05:55:13.405229 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.405215 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 05:55:13.416197 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.416180 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 05:55:13.416452 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.416439 4855 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 05:55:13.416503 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.416482 4855 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-67.ec2.internal" Nov 03 05:55:13.416543 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.416441 4855 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 05:55:13.416577 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.416567 4855 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-67.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-67.ec2.internal" Nov 03 05:55:13.416753 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.416736 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 05:55:13.454946 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.454925 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.455022 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.454953 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/56b77a5b-a55e-4849-9789-b9c5a3db4b02-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:13.455022 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.454970 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-serving-cert\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:13.455022 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.454986 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-ca-trust-extracted\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.455022 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455005 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-bound-sa-token\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.455194 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455130 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-ovn\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.455194 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455167 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.455194 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455178 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-serving-cert\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:13.455338 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455198 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-socket-dir-parent\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.455338 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455230 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pnq\" (UniqueName: \"kubernetes.io/projected/1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec-kube-api-access-t5pnq\") pod \"network-check-source-6f78d59f99-n9hcn\" (UID: \"1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn" Nov 03 05:55:13.455338 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455257 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:13.455338 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455299 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-ca-trust-extracted\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.455562 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455354 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-installation-pull-secrets\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.455562 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455371 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/56b77a5b-a55e-4849-9789-b9c5a3db4b02-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:13.455562 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455406 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:13.455562 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455451 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlrf\" (UniqueName: \"kubernetes.io/projected/1eb02066-c20a-45b3-885a-0e9834b1c998-kube-api-access-xhlrf\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:55:13.455562 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455480 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c48a6b-a95c-4c64-9d89-1ed9d0320908-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:13.455562 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455513 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf7vt\" (UniqueName: \"kubernetes.io/projected/059a8188-2f82-4b0d-af82-5598a6e7b50e-kube-api-access-tf7vt\") pod \"ingress-canary-6j7pt\" (UID: \"059a8188-2f82-4b0d-af82-5598a6e7b50e\") " pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:13.455562 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455520 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:13.455562 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455539 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-cnibin\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.455562 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455561 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-kubelet-config\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455581 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1eb02066-c20a-45b3-885a-0e9834b1c998-serviceca\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455601 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsz5c\" (UniqueName: \"kubernetes.io/projected/58637e90-c0b9-44da-91e9-7e1075a2636e-kube-api-access-gsz5c\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455613 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-installation-pull-secrets\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455622 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455644 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp2g4\" (UniqueName: \"kubernetes.io/projected/054b9d40-fa74-4f73-9db4-70ff0dd43191-kube-api-access-zp2g4\") pod \"kas-connection-checker-685895c7c9-tnzjq\" (UID: \"054b9d40-fa74-4f73-9db4-70ff0dd43191\") " pod="kube-system/kas-connection-checker-685895c7c9-tnzjq" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455671 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a09f2-9d41-40f6-94f1-6cb00971d950-config-volume\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455687 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c48a6b-a95c-4c64-9d89-1ed9d0320908-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455695 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-run\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455735 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d73ab4e1-b560-420b-8216-6272bd184f39-original-pull-secret\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455755 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-tuned\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455847 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1eb02066-c20a-45b3-885a-0e9834b1c998-serviceca\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455895 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8z96\" (UniqueName: \"kubernetes.io/projected/76c09b47-0ab3-4212-924b-131d88e94ab3-kube-api-access-q8z96\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455903 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a09f2-9d41-40f6-94f1-6cb00971d950-config-volume\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:13.455920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455907 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-tuned\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.456377 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.455966 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-device-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.456377 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456073 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-service-ca-bundle\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.456377 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456108 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjm42\" (UniqueName: \"kubernetes.io/projected/36beafc8-4aee-448b-8dfd-64f89637aa9f-kube-api-access-cjm42\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:55:13.456377 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456125 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-multus-certs\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.456377 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456214 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-service-ca-bundle\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.456377 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456216 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:13.456377 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456311 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.456377 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456335 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2104714e-6924-4133-90b9-7d98135901e8-iptables-alerter-script\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:55:13.456377 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456365 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-config\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:13.456751 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456426 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:55:13.456751 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456536 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-etc-kubernetes\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.456751 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456581 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwqc\" (UniqueName: \"kubernetes.io/projected/5732cb6f-524c-4389-8fe2-f81885322ae5-kube-api-access-2mwqc\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.456751 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456581 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-config\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:13.456751 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456621 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2104714e-6924-4133-90b9-7d98135901e8-iptables-alerter-script\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:55:13.456751 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456695 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-certificates\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.456751 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456733 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-system-cni-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.457019 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456840 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fm7p\" (UniqueName: \"kubernetes.io/projected/db236f0a-264d-4498-8dd1-1787e2a3ee95-kube-api-access-5fm7p\") pod \"kas-connection-checker-685895c7c9-bhkn5\" (UID: \"db236f0a-264d-4498-8dd1-1787e2a3ee95\") " pod="kube-system/kas-connection-checker-685895c7c9-bhkn5" Nov 03 05:55:13.457019 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456840 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-certificates\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.457019 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456885 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.457019 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.456995 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-netd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.457171 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457038 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08a92738-2cd0-4eed-9e83-c01baed110c2-data-volume\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.457171 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457074 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkw2l\" (UniqueName: \"kubernetes.io/projected/99c39ff9-f070-4c29-8284-4f98d116faa4-kube-api-access-lkw2l\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:13.457171 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457100 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08a92738-2cd0-4eed-9e83-c01baed110c2-data-volume\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.457171 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457123 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znfx\" (UniqueName: \"kubernetes.io/projected/9ef729d9-2e44-4c39-a018-5df7a941dcc5-kube-api-access-4znfx\") pod \"kas-connection-checker-685895c7c9-6z5gg\" (UID: \"9ef729d9-2e44-4c39-a018-5df7a941dcc5\") " pod="kube-system/kas-connection-checker-685895c7c9-6z5gg" Nov 03 05:55:13.457171 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457152 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-installation-pull-secrets\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.457364 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457180 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-trusted-ca\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:13.457364 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457343 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-installation-pull-secrets\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.457489 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457367 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.457489 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457406 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-lib-modules\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.457489 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457425 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-dbus\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:13.457489 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457441 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1eb02066-c20a-45b3-885a-0e9834b1c998-host\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:55:13.457489 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457457 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:13.457698 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457491 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-etc-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.457698 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457514 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-cni-binary-copy\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.457698 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457530 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-binary-copy\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.457698 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457563 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.457698 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457583 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-config\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:13.457698 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457587 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-trusted-ca\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:13.457698 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457601 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b099592-879f-4313-be96-60d39b44f8a8-serving-cert\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:13.457698 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457634 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457727 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktj9\" (UniqueName: \"kubernetes.io/projected/42da3e8e-febb-4e95-ae4c-c750c4bd3481-kube-api-access-wktj9\") pod \"volume-data-source-validator-58c9f5f46d-74hnk\" (UID: \"42da3e8e-febb-4e95-ae4c-c750c4bd3481\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457741 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-binary-copy\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457758 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-image-registry-private-configuration\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457786 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457791 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b099592-879f-4313-be96-60d39b44f8a8-config\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457814 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67xpb\" (UniqueName: \"kubernetes.io/projected/893a8c11-60f6-4736-9971-10520459a1d0-kube-api-access-67xpb\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457859 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-cni-binary-copy\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457868 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4tsh\" (UniqueName: \"kubernetes.io/projected/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-kube-api-access-k4tsh\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457880 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b099592-879f-4313-be96-60d39b44f8a8-serving-cert\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457902 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-conf\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457933 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74a0e103-b720-45e0-8f30-c54e9542b5cf-ca-trust-extracted\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457957 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/315726a9-3009-40f6-bf65-6818acd95046-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.457980 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-kubelet\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458004 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74a0e103-b720-45e0-8f30-c54e9542b5cf-ca-trust-extracted\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458004 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-image-registry-private-configuration\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458010 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.458065 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458057 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-trusted-ca\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.458730 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458107 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-default-certificate\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.458730 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458506 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-bin\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.458730 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458564 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:13.458730 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458571 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-default-certificate\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.458730 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458591 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-hostroot\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.458730 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458645 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-trusted-ca\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.458730 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458704 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmvr6\" (UniqueName: \"kubernetes.io/projected/315726a9-3009-40f6-bf65-6818acd95046-kube-api-access-vmvr6\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.459020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458741 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-netns\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.459020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458772 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89mmd\" (UniqueName: \"kubernetes.io/projected/0b722d0a-df66-409f-a7cf-0a891f5a1d32-kube-api-access-89mmd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.459020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458801 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.459020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458833 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-multus-daemon-config\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.459020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458860 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5g9x\" (UniqueName: \"kubernetes.io/projected/80c48a6b-a95c-4c64-9d89-1ed9d0320908-kube-api-access-g5g9x\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:13.459020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458891 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-tmp\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.459020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458922 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.459020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458952 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-tmp\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.459020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.458999 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4ch\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-kube-api-access-gl4ch\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.459461 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459114 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-tmp\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.459461 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459245 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.459461 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459308 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f174941-0801-466a-8cef-c2da4faa916f-multus-daemon-config\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.459614 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459507 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-node-log\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.459614 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459566 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7502a388-bbcd-4916-8b8d-a4258c58d575-agent-certs\") pod \"konnectivity-agent-sp2qh\" (UID: \"7502a388-bbcd-4916-8b8d-a4258c58d575\") " pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:55:13.459614 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459602 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-conf-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.459741 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459661 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85bfd499-9f29-4d6e-b2de-094d011b4f0b-tmp\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.459741 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459700 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-sys\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.459741 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459735 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-var-lib-kubelet\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.459862 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459773 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-snapshots\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.459862 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459836 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgfp\" (UniqueName: \"kubernetes.io/projected/56b77a5b-a55e-4849-9789-b9c5a3db4b02-kube-api-access-dqgfp\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:13.459947 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459868 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqbfm\" (UniqueName: \"kubernetes.io/projected/2f174941-0801-466a-8cef-c2da4faa916f-kube-api-access-cqbfm\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.459947 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459900 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80c48a6b-a95c-4c64-9d89-1ed9d0320908-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:13.459947 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459934 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-kubernetes\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.460073 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.459967 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxwvv\" (UniqueName: \"kubernetes.io/projected/85bfd499-9f29-4d6e-b2de-094d011b4f0b-kube-api-access-jxwvv\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.460073 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460061 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.460154 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460097 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.460154 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460123 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7502a388-bbcd-4916-8b8d-a4258c58d575-agent-certs\") pod \"konnectivity-agent-sp2qh\" (UID: \"7502a388-bbcd-4916-8b8d-a4258c58d575\") " pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:55:13.460154 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460133 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-env-overrides\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.460276 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460160 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysconfig\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.460276 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460191 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-host\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.460276 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460221 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.460276 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460259 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-script-lib\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.460456 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460292 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-kubelet\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.460456 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460424 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36beafc8-4aee-448b-8dfd-64f89637aa9f-tmp-dir\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:55:13.460539 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460464 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.460539 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460497 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-stats-auth\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.460539 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460528 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-slash\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.460668 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460555 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-systemd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.460668 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460578 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-log-socket\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.460668 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460601 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-script-lib\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.460668 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460608 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-config\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.460668 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460640 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-system-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.460930 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460760 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/315726a9-3009-40f6-bf65-6818acd95046-snapshots\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.460930 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460816 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-bound-sa-token\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.460930 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460845 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7hfp\" (UniqueName: \"kubernetes.io/projected/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-api-access-k7hfp\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.460930 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460879 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-cnibin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.460930 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460902 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovnkube-config\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.460930 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460912 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-modprobe-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.461168 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460948 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-image-registry-private-configuration\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.461168 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.460978 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovn-node-metrics-cert\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.461168 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461013 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08a92738-2cd0-4eed-9e83-c01baed110c2-crio-socket\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.461168 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461042 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-bin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.461168 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461075 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-864s8\" (UniqueName: \"kubernetes.io/projected/bd3a09f2-9d41-40f6-94f1-6cb00971d950-kube-api-access-864s8\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:13.461372 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461308 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80c48a6b-a95c-4c64-9d89-1ed9d0320908-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:13.461430 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461376 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-systemd\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.461488 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461431 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-var-lib-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.461488 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461471 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr54g\" (UniqueName: \"kubernetes.io/projected/2b099592-879f-4313-be96-60d39b44f8a8-kube-api-access-nr54g\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:13.461577 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461503 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:13.461577 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461530 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.461577 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461538 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-image-registry-private-configuration\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.461577 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461562 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.461744 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461595 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-multus\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.461744 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461628 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-trusted-ca\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.461744 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461658 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzl4\" (UniqueName: \"kubernetes.io/projected/2104714e-6924-4133-90b9-7d98135901e8-kube-api-access-5jzl4\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:55:13.461744 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461690 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-systemd-units\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.461744 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461720 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.461744 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461716 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b722d0a-df66-409f-a7cf-0a891f5a1d32-ovn-node-metrics-cert\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.461988 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.461764 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:13.462411 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.462120 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b722d0a-df66-409f-a7cf-0a891f5a1d32-env-overrides\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.462411 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.462195 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36beafc8-4aee-448b-8dfd-64f89637aa9f-tmp-dir\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:55:13.462411 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.462274 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:13.462578 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.462431 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-stats-auth\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.462792 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.462766 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.462792 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.462782 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-trusted-ca\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.463442 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.463421 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79lwr\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-kube-api-access-79lwr\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.463593 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.463573 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.464054 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464019 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-netns\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.464121 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464075 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd3a09f2-9d41-40f6-94f1-6cb00971d950-tmp-dir\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:13.464180 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464128 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/315726a9-3009-40f6-bf65-6818acd95046-serving-cert\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.464180 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464166 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36beafc8-4aee-448b-8dfd-64f89637aa9f-hosts-file\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:55:13.464277 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464180 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd3a09f2-9d41-40f6-94f1-6cb00971d950-tmp-dir\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:13.464277 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464190 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5732cb6f-524c-4389-8fe2-f81885322ae5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.464370 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464273 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-certificates\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.464370 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464298 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/315726a9-3009-40f6-bf65-6818acd95046-serving-cert\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.464370 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464323 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7502a388-bbcd-4916-8b8d-a4258c58d575-konnectivity-ca\") pod \"konnectivity-agent-sp2qh\" (UID: \"7502a388-bbcd-4916-8b8d-a4258c58d575\") " pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:55:13.464542 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464383 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-k8s-cni-cncf-io\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.464542 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464491 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-certificates\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.464542 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464512 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/059a8188-2f82-4b0d-af82-5598a6e7b50e-cert\") pod \"ingress-canary-6j7pt\" (UID: \"059a8188-2f82-4b0d-af82-5598a6e7b50e\") " pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:13.464686 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464554 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2104714e-6924-4133-90b9-7d98135901e8-host-slash\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:55:13.464686 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464563 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7502a388-bbcd-4916-8b8d-a4258c58d575-konnectivity-ca\") pod \"konnectivity-agent-sp2qh\" (UID: \"7502a388-bbcd-4916-8b8d-a4258c58d575\") " pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:55:13.464773 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.464722 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/059a8188-2f82-4b0d-af82-5598a6e7b50e-cert\") pod \"ingress-canary-6j7pt\" (UID: \"059a8188-2f82-4b0d-af82-5598a6e7b50e\") " pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:13.479400 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.479377 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-bound-sa-token\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.480575 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.480561 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlrf\" (UniqueName: \"kubernetes.io/projected/1eb02066-c20a-45b3-885a-0e9834b1c998-kube-api-access-xhlrf\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:55:13.500309 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.500280 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp2g4\" (UniqueName: \"kubernetes.io/projected/054b9d40-fa74-4f73-9db4-70ff0dd43191-kube-api-access-zp2g4\") pod \"kas-connection-checker-685895c7c9-tnzjq\" (UID: \"054b9d40-fa74-4f73-9db4-70ff0dd43191\") " pod="kube-system/kas-connection-checker-685895c7c9-tnzjq" Nov 03 05:55:13.523601 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.523550 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwqc\" (UniqueName: \"kubernetes.io/projected/5732cb6f-524c-4389-8fe2-f81885322ae5-kube-api-access-2mwqc\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.523601 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.523585 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjm42\" (UniqueName: \"kubernetes.io/projected/36beafc8-4aee-448b-8dfd-64f89637aa9f-kube-api-access-cjm42\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:55:13.523766 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.523662 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znfx\" (UniqueName: \"kubernetes.io/projected/9ef729d9-2e44-4c39-a018-5df7a941dcc5-kube-api-access-4znfx\") pod \"kas-connection-checker-685895c7c9-6z5gg\" (UID: \"9ef729d9-2e44-4c39-a018-5df7a941dcc5\") " pod="kube-system/kas-connection-checker-685895c7c9-6z5gg" Nov 03 05:55:13.523766 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.523736 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pnq\" (UniqueName: \"kubernetes.io/projected/1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec-kube-api-access-t5pnq\") pod \"network-check-source-6f78d59f99-n9hcn\" (UID: \"1a38e8c6-468e-4a7f-8804-fde7ea3bb3ec\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-n9hcn" Nov 03 05:55:13.524313 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.524292 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktj9\" (UniqueName: \"kubernetes.io/projected/42da3e8e-febb-4e95-ae4c-c750c4bd3481-kube-api-access-wktj9\") pod \"volume-data-source-validator-58c9f5f46d-74hnk\" (UID: \"42da3e8e-febb-4e95-ae4c-c750c4bd3481\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-74hnk" Nov 03 05:55:13.524344 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.524300 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmvr6\" (UniqueName: \"kubernetes.io/projected/315726a9-3009-40f6-bf65-6818acd95046-kube-api-access-vmvr6\") pod \"insights-operator-7c4986d449-86pnb\" (UID: \"315726a9-3009-40f6-bf65-6818acd95046\") " pod="openshift-insights/insights-operator-7c4986d449-86pnb" Nov 03 05:55:13.537182 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.537125 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxwvv\" (UniqueName: \"kubernetes.io/projected/85bfd499-9f29-4d6e-b2de-094d011b4f0b-kube-api-access-jxwvv\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.537256 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.537242 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf7vt\" (UniqueName: \"kubernetes.io/projected/059a8188-2f82-4b0d-af82-5598a6e7b50e-kube-api-access-tf7vt\") pod \"ingress-canary-6j7pt\" (UID: \"059a8188-2f82-4b0d-af82-5598a6e7b50e\") " pod="openshift-ingress-canary/ingress-canary-6j7pt" Nov 03 05:55:13.542020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.542001 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fm7p\" (UniqueName: \"kubernetes.io/projected/db236f0a-264d-4498-8dd1-1787e2a3ee95-kube-api-access-5fm7p\") pod \"kas-connection-checker-685895c7c9-bhkn5\" (UID: \"db236f0a-264d-4498-8dd1-1787e2a3ee95\") " pod="kube-system/kas-connection-checker-685895c7c9-bhkn5" Nov 03 05:55:13.542114 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.542097 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzl4\" (UniqueName: \"kubernetes.io/projected/2104714e-6924-4133-90b9-7d98135901e8-kube-api-access-5jzl4\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:55:13.542581 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.542567 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-864s8\" (UniqueName: \"kubernetes.io/projected/bd3a09f2-9d41-40f6-94f1-6cb00971d950-kube-api-access-864s8\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:13.542807 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.542787 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8z96\" (UniqueName: \"kubernetes.io/projected/76c09b47-0ab3-4212-924b-131d88e94ab3-kube-api-access-q8z96\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.543031 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.543017 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79lwr\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-kube-api-access-79lwr\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.544177 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.544157 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsz5c\" (UniqueName: \"kubernetes.io/projected/58637e90-c0b9-44da-91e9-7e1075a2636e-kube-api-access-gsz5c\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:13.544462 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.544445 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5g9x\" (UniqueName: \"kubernetes.io/projected/80c48a6b-a95c-4c64-9d89-1ed9d0320908-kube-api-access-g5g9x\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7\" (UID: \"80c48a6b-a95c-4c64-9d89-1ed9d0320908\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" Nov 03 05:55:13.544571 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.544552 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr54g\" (UniqueName: \"kubernetes.io/projected/2b099592-879f-4313-be96-60d39b44f8a8-kube-api-access-nr54g\") pod \"console-operator-7cd56b8dbc-qwvxg\" (UID: \"2b099592-879f-4313-be96-60d39b44f8a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:13.545014 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.545000 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl4ch\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-kube-api-access-gl4ch\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.545210 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.545195 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-bound-sa-token\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.547440 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.547425 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89mmd\" (UniqueName: \"kubernetes.io/projected/0b722d0a-df66-409f-a7cf-0a891f5a1d32-kube-api-access-89mmd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.548034 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.548020 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwsq8\" (UniqueName: \"kubernetes.io/projected/219a457b-c7b8-4a1d-8f14-50a5d56bb9bd-kube-api-access-cwsq8\") pod \"network-check-target-cp5xp\" (UID: \"219a457b-c7b8-4a1d-8f14-50a5d56bb9bd\") " pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:55:13.548456 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.548439 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4tsh\" (UniqueName: \"kubernetes.io/projected/49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f-kube-api-access-k4tsh\") pod \"service-ca-operator-544d749c8c-g27mq\" (UID: \"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" Nov 03 05:55:13.549186 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.549171 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgfp\" (UniqueName: \"kubernetes.io/projected/56b77a5b-a55e-4849-9789-b9c5a3db4b02-kube-api-access-dqgfp\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:13.549410 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.549377 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7hfp\" (UniqueName: \"kubernetes.io/projected/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-api-access-k7hfp\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.549639 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.549625 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqbfm\" (UniqueName: \"kubernetes.io/projected/2f174941-0801-466a-8cef-c2da4faa916f-kube-api-access-cqbfm\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.549722 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.549707 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67xpb\" (UniqueName: \"kubernetes.io/projected/893a8c11-60f6-4736-9971-10520459a1d0-kube-api-access-67xpb\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.553226 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.553210 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkw2l\" (UniqueName: \"kubernetes.io/projected/99c39ff9-f070-4c29-8284-4f98d116faa4-kube-api-access-lkw2l\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:13.565596 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565579 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:13.565660 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565604 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.565660 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565619 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-conf\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.565660 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565634 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-kubelet\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.565660 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565648 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565663 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-bin\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565678 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565693 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565697 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-hostroot\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565733 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-netns\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565734 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565738 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-bin\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565759 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-conf\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565695 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-kubelet\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565760 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565792 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-netns\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565819 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-node-log\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565843 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-hostroot\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.565845 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565849 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-conf-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.565824 4855 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565870 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-node-log\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565892 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-conf-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.565831 4855 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.565932 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls podName:56b77a5b-a55e-4849-9789-b9c5a3db4b02 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.065912783 +0000 UTC m=+1.874300049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vpktf" (UID: "56b77a5b-a55e-4849-9789-b9c5a3db4b02") : secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565948 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-sys\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.565976 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.0659591 +0000 UTC m=+1.874346359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : secret "insights-runtime-extractor-tls" not found Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565986 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-sys\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.565915 4855 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.565995 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-var-lib-kubelet\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566035 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-kubernetes\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.566066 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls podName:99c39ff9-f070-4c29-8284-4f98d116faa4 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.066054628 +0000 UTC m=+1.874441874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d277j" (UID: "99c39ff9-f070-4c29-8284-4f98d116faa4") : secret "samples-operator-tls" not found Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566091 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566102 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-kubernetes\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566111 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566100 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-var-lib-kubelet\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566137 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysconfig\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566164 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-host\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566179 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysconfig\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566183 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566204 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-kubelet\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566208 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566228 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566231 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-kubelet\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566245 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-slash\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566267 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-systemd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566269 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-host\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566266 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566287 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-log-socket\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.566289 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566302 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-system-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566338 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-slash\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566343 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-system-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566364 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-cnibin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566364 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566378 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-systemd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.566410 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.066382601 +0000 UTC m=+1.874769847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : configmap references non-existent config key: service-ca.crt Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566380 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-modprobe-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566427 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-log-socket\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566461 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-cnibin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566499 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08a92738-2cd0-4eed-9e83-c01baed110c2-crio-socket\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566521 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-modprobe-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566547 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08a92738-2cd0-4eed-9e83-c01baed110c2-crio-socket\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566557 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-bin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566574 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-systemd\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566576 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-bin\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566589 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-var-lib-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566604 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-systemd\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566604 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566629 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566646 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-var-lib-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566650 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-multus\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566667 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-var-lib-cni-multus\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566672 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-etc-sysctl-d\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566692 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-systemd-units\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.566719 4855 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.566727 4855 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c96d9b8b6-5hh78: secret "image-registry-tls" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566745 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566717 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.566757 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls podName:74a0e103-b720-45e0-8f30-c54e9542b5cf nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.066748862 +0000 UTC m=+1.875136108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls") pod "image-registry-7c96d9b8b6-5hh78" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf") : secret "image-registry-tls" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566782 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566801 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566815 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-netns\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566783 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-systemd-units\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566832 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36beafc8-4aee-448b-8dfd-64f89637aa9f-hosts-file\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.566842 4855 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566856 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-netns\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566853 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-k8s-cni-cncf-io\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566876 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2104714e-6924-4133-90b9-7d98135901e8-host-slash\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566880 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.566892 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert podName:8ba1e446-7be2-4673-9a44-a6c6a5e1be2d nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.066877856 +0000 UTC m=+1.875265102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zknxx" (UID: "8ba1e446-7be2-4673-9a44-a6c6a5e1be2d") : secret "networking-console-plugin-cert" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566900 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2104714e-6924-4133-90b9-7d98135901e8-host-slash\") pod \"iptables-alerter-xzjql\" (UID: \"2104714e-6924-4133-90b9-7d98135901e8\") " pod="openshift-network-operator/iptables-alerter-xzjql" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566890 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36beafc8-4aee-448b-8dfd-64f89637aa9f-hosts-file\") pod \"node-resolver-99n4t\" (UID: \"36beafc8-4aee-448b-8dfd-64f89637aa9f\") " pod="openshift-dns/node-resolver-99n4t" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566920 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566910 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-k8s-cni-cncf-io\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566940 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-ovn\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566955 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566973 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-socket-dir-parent\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.566991 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-run-ovn\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567001 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-cnibin\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567023 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567026 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-kubelet-config\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567052 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567075 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-run\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567080 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-cnibin\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567085 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-kubelet-config\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567097 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-device-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.567107 4855 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.567116 4855 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67898f7b5f-gmmrp: secret "image-registry-tls" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567125 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-multus-certs\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.567143 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls podName:f29d1b5f-b1e2-45e0-8b1f-da929d562b71 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.067135002 +0000 UTC m=+1.875522273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls") pod "image-registry-67898f7b5f-gmmrp" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71") : secret "image-registry-tls" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567139 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-run\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567160 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/893a8c11-60f6-4736-9971-10520459a1d0-device-dir\") pod \"aws-ebs-csi-driver-node-cwkhg\" (UID: \"893a8c11-60f6-4736-9971-10520459a1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkhg" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567156 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567172 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-host-run-multus-certs\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567056 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-socket-dir-parent\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.567196 4855 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567218 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.567260 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls podName:bd3a09f2-9d41-40f6-94f1-6cb00971d950 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.067244804 +0000 UTC m=+1.875632065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls") pod "dns-default-g9htv" (UID: "bd3a09f2-9d41-40f6-94f1-6cb00971d950") : secret "dns-default-metrics-tls" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567319 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567319 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-etc-kubernetes\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567343 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-system-cni-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567351 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-etc-kubernetes\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567358 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567384 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-netd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567422 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5732cb6f-524c-4389-8fe2-f81885322ae5-system-cni-dir\") pod \"multus-additional-cni-plugins-9c7rn\" (UID: \"5732cb6f-524c-4389-8fe2-f81885322ae5\") " pod="openshift-multus/multus-additional-cni-plugins-9c7rn" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567429 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567449 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-host-cni-netd\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.567434 4855 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567456 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08a92738-2cd0-4eed-9e83-c01baed110c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567453 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-lib-modules\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567473 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f174941-0801-466a-8cef-c2da4faa916f-multus-cni-dir\") pod \"multus-w4bpx\" (UID: \"2f174941-0801-466a-8cef-c2da4faa916f\") " pod="openshift-multus/multus-w4bpx" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.567479 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.067471667 +0000 UTC m=+1.875858947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : secret "router-metrics-certs-default" not found Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567519 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-dbus\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567537 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1eb02066-c20a-45b3-885a-0e9834b1c998-host\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567540 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85bfd499-9f29-4d6e-b2de-094d011b4f0b-lib-modules\") pod \"tuned-kkvkc\" (UID: \"85bfd499-9f29-4d6e-b2de-094d011b4f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-kkvkc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567557 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d73ab4e1-b560-420b-8216-6272bd184f39-dbus\") pod \"global-pull-secret-syncer-vspq2\" (UID: \"d73ab4e1-b560-420b-8216-6272bd184f39\") " pod="kube-system/global-pull-secret-syncer-vspq2" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567579 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1eb02066-c20a-45b3-885a-0e9834b1c998-host\") pod \"node-ca-vvkcc\" (UID: \"1eb02066-c20a-45b3-885a-0e9834b1c998\") " pod="openshift-image-registry/node-ca-vvkcc" Nov 03 05:55:13.567551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567602 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:13.569942 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567627 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-etc-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.569942 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.567695 4855 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 05:55:13.569942 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:13.567723 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.067714526 +0000 UTC m=+1.876101772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : secret "metrics-daemon-secret" not found Nov 03 05:55:13.569942 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.567694 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b722d0a-df66-409f-a7cf-0a891f5a1d32-etc-openvswitch\") pod \"ovnkube-node-c75h9\" (UID: \"0b722d0a-df66-409f-a7cf-0a891f5a1d32\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:13.597858 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.597835 4855 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sp2qh_7502a388-bbcd-4916-8b8d-a4258c58d575/konnectivity-agent/0.log" limit=1024 Nov 03 05:55:13.653828 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:13.653811 4855 scope.go:122] "RemoveContainer" containerID="a4debb86934e6ea1da15236ab15df42b19e3ca5cef0906b0ce4d4abfd1f63885" Nov 03 05:55:14.071453 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.071368 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:14.071453 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.071441 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:14.071675 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.071473 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:14.071675 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071520 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:15.071504874 +0000 UTC m=+2.879892143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : configmap references non-existent config key: service-ca.crt Nov 03 05:55:14.071675 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.071563 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:14.071675 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071576 4855 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 05:55:14.071675 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071580 4855 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:14.071675 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071600 4855 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c96d9b8b6-5hh78: secret "image-registry-tls" not found Nov 03 05:55:14.071675 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071625 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert podName:8ba1e446-7be2-4673-9a44-a6c6a5e1be2d nodeName:}" failed. No retries permitted until 2026-11-03 05:55:15.07160959 +0000 UTC m=+2.879996854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zknxx" (UID: "8ba1e446-7be2-4673-9a44-a6c6a5e1be2d") : secret "networking-console-plugin-cert" not found Nov 03 05:55:14.071675 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.071599 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:14.071675 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071643 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls podName:74a0e103-b720-45e0-8f30-c54e9542b5cf nodeName:}" failed. No retries permitted until 2026-11-03 05:55:15.071633954 +0000 UTC m=+2.880021208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls") pod "image-registry-7c96d9b8b6-5hh78" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf") : secret "image-registry-tls" not found Nov 03 05:55:14.071675 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071666 4855 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071711 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls podName:bd3a09f2-9d41-40f6-94f1-6cb00971d950 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:15.071698297 +0000 UTC m=+2.880085580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls") pod "dns-default-g9htv" (UID: "bd3a09f2-9d41-40f6-94f1-6cb00971d950") : secret "dns-default-metrics-tls" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.071689 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071769 4855 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071805 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:15.071794545 +0000 UTC m=+2.880181813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : secret "router-metrics-certs-default" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.071766 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071825 4855 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.071854 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071865 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:55:15.071851874 +0000 UTC m=+2.880239149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : secret "metrics-daemon-secret" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071911 4855 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.071912 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071951 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls podName:99c39ff9-f070-4c29-8284-4f98d116faa4 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:15.071938447 +0000 UTC m=+2.880325711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d277j" (UID: "99c39ff9-f070-4c29-8284-4f98d116faa4") : secret "samples-operator-tls" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.071970 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.071986 4855 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.072026 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls podName:56b77a5b-a55e-4849-9789-b9c5a3db4b02 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:15.072015329 +0000 UTC m=+2.880402606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vpktf" (UID: "56b77a5b-a55e-4849-9789-b9c5a3db4b02") : secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.072076 4855 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.072082 4855 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.072095 4855 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67898f7b5f-gmmrp: secret "image-registry-tls" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.072116 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:15.072104871 +0000 UTC m=+2.880492120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : secret "insights-runtime-extractor-tls" not found Nov 03 05:55:14.072161 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.072141 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls podName:f29d1b5f-b1e2-45e0-8b1f-da929d562b71 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:15.072132956 +0000 UTC m=+2.880520205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls") pod "image-registry-67898f7b5f-gmmrp" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71") : secret "image-registry-tls" not found Nov 03 05:55:14.383914 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.383844 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 05:55:14.384306 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.384247 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/1.log" Nov 03 05:55:14.384306 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.384276 4855 generic.go:356] "Generic (PLEG): container finished" podID="2b099592-879f-4313-be96-60d39b44f8a8" containerID="2278415587d3978e22ebfe07d162a1060d19e7a8bc4bad398af0e40dd5f9ab60" exitCode=255 Nov 03 05:55:14.384409 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.384362 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" event={"ID":"2b099592-879f-4313-be96-60d39b44f8a8","Type":"ContainerDied","Data":"2278415587d3978e22ebfe07d162a1060d19e7a8bc4bad398af0e40dd5f9ab60"} Nov 03 05:55:14.384462 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.384446 4855 scope.go:122] "RemoveContainer" containerID="a4debb86934e6ea1da15236ab15df42b19e3ca5cef0906b0ce4d4abfd1f63885" Nov 03 05:55:14.384748 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:14.384726 4855 scope.go:122] "RemoveContainer" containerID="2278415587d3978e22ebfe07d162a1060d19e7a8bc4bad398af0e40dd5f9ab60" Nov 03 05:55:14.384932 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:14.384913 4855 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qwvxg_openshift-console-operator(2b099592-879f-4313-be96-60d39b44f8a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" podUID="2b099592-879f-4313-be96-60d39b44f8a8" Nov 03 05:55:15.080867 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.080829 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.080873 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.080921 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.080951 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.080980 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.080991 4855 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081011 4855 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c96d9b8b6-5hh78: secret "image-registry-tls" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.081013 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.081041 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081054 4855 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081072 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls podName:74a0e103-b720-45e0-8f30-c54e9542b5cf nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.081054386 +0000 UTC m=+4.889441646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls") pod "image-registry-7c96d9b8b6-5hh78" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf") : secret "image-registry-tls" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081079 4855 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081081 4855 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081110 4855 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.081108 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081120 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert podName:8ba1e446-7be2-4673-9a44-a6c6a5e1be2d nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.081103812 +0000 UTC m=+4.889491073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zknxx" (UID: "8ba1e446-7be2-4673-9a44-a6c6a5e1be2d") : secret "networking-console-plugin-cert" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081121 4855 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081143 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.081134841 +0000 UTC m=+4.889522087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : secret "router-metrics-certs-default" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081062 4855 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:15.081152 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081165 4855 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081171 4855 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67898f7b5f-gmmrp: secret "image-registry-tls" not found Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081160 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls podName:bd3a09f2-9d41-40f6-94f1-6cb00971d950 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.081152553 +0000 UTC m=+4.889539801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls") pod "dns-default-g9htv" (UID: "bd3a09f2-9d41-40f6-94f1-6cb00971d950") : secret "dns-default-metrics-tls" not found Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081218 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls podName:99c39ff9-f070-4c29-8284-4f98d116faa4 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.081201876 +0000 UTC m=+4.889589121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d277j" (UID: "99c39ff9-f070-4c29-8284-4f98d116faa4") : secret "samples-operator-tls" not found Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081233 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.081226071 +0000 UTC m=+4.889613317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : secret "metrics-daemon-secret" not found Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081252 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls podName:56b77a5b-a55e-4849-9789-b9c5a3db4b02 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.081247141 +0000 UTC m=+4.889634387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vpktf" (UID: "56b77a5b-a55e-4849-9789-b9c5a3db4b02") : secret "cluster-monitoring-operator-tls" not found Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081262 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls podName:f29d1b5f-b1e2-45e0-8b1f-da929d562b71 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.081257675 +0000 UTC m=+4.889644921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls") pod "image-registry-67898f7b5f-gmmrp" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71") : secret "image-registry-tls" not found Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.081279 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.081316 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081414 4855 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081445 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.081437944 +0000 UTC m=+4.889825191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : configmap references non-existent config key: service-ca.crt Nov 03 05:55:15.081839 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:15.081469 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls podName:08a92738-2cd0-4eed-9e83-c01baed110c2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:17.081452483 +0000 UTC m=+4.889839749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqt2f" (UID: "08a92738-2cd0-4eed-9e83-c01baed110c2") : secret "insights-runtime-extractor-tls" not found Nov 03 05:55:15.388573 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:15.388502 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 05:55:16.406909 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:16.406876 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:17.100764 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.100728 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:17.100976 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.100770 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:17.100976 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.100802 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:17.100976 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.100831 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:17.100976 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.100849 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:17.100976 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.100877 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:17.100976 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.100904 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:17.100976 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.100937 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:17.100976 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.100968 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.100993 4855 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101010 4855 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101030 4855 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67898f7b5f-gmmrp: secret "image-registry-tls" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101037 4855 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.100995 4855 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101059 4855 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c96d9b8b6-5hh78: secret "image-registry-tls" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101005 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:21.100982078 +0000 UTC m=+8.909369343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : configmap references non-existent config key: service-ca.crt Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101047 4855 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101151 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert podName:8ba1e446-7be2-4673-9a44-a6c6a5e1be2d nodeName:}" failed. No retries permitted until 2026-11-03 05:55:21.101134577 +0000 UTC m=+8.909521827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zknxx" (UID: "8ba1e446-7be2-4673-9a44-a6c6a5e1be2d") : secret "networking-console-plugin-cert" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101167 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls podName:f29d1b5f-b1e2-45e0-8b1f-da929d562b71 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:21.101159362 +0000 UTC m=+8.909546621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls") pod "image-registry-67898f7b5f-gmmrp" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71") : secret "image-registry-tls" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101177 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:55:21.101171692 +0000 UTC m=+8.909558938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : secret "metrics-daemon-secret" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101186 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls podName:74a0e103-b720-45e0-8f30-c54e9542b5cf nodeName:}" failed. No retries permitted until 2026-11-03 05:55:21.101182305 +0000 UTC m=+8.909569551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls") pod "image-registry-7c96d9b8b6-5hh78" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf") : secret "image-registry-tls" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:17.101196 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls podName:bd3a09f2-9d41-40f6-94f1-6cb00971d950 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:21.101191495 +0000 UTC m=+8.909578741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls") pod "dns-default-g9htv" (UID: "bd3a09f2-9d41-40f6-94f1-6cb00971d950") : secret "dns-default-metrics-tls" not found Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.101190 4855 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 05:55:17.101375 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.101214 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:17.103552 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.103534 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/56b77a5b-a55e-4849-9789-b9c5a3db4b02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vpktf\" (UID: \"56b77a5b-a55e-4849-9789-b9c5a3db4b02\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:17.103649 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.103568 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08a92738-2cd0-4eed-9e83-c01baed110c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqt2f\" (UID: \"08a92738-2cd0-4eed-9e83-c01baed110c2\") " pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:17.103649 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.103571 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/99c39ff9-f070-4c29-8284-4f98d116faa4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d277j\" (UID: \"99c39ff9-f070-4c29-8284-4f98d116faa4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:17.103715 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.103674 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c09b47-0ab3-4212-924b-131d88e94ab3-metrics-certs\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:17.253139 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.253109 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" Nov 03 05:55:17.254172 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.254154 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" Nov 03 05:55:17.264507 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.264484 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tqt2f" Nov 03 05:55:17.557355 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.557322 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j"] Nov 03 05:55:17.571654 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.571514 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf"] Nov 03 05:55:17.576488 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:17.576459 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b77a5b_a55e_4849_9789_b9c5a3db4b02.slice/crio-d9a1aeeea93e92b314bff3ddfa976a74eaf0c10a161a6af051b3721947205f10 WatchSource:0}: Error finding container d9a1aeeea93e92b314bff3ddfa976a74eaf0c10a161a6af051b3721947205f10: Status 404 returned error can't find the container with id d9a1aeeea93e92b314bff3ddfa976a74eaf0c10a161a6af051b3721947205f10 Nov 03 05:55:17.578440 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.578422 4855 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 05:55:17.629046 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.629022 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tqt2f"] Nov 03 05:55:17.633765 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:17.633732 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a92738_2cd0_4eed_9e83_c01baed110c2.slice/crio-2bddb93b6c7ea30c1a865c5e32f4b692a32343b06b6bc4d52d7ecb333e2fc355 WatchSource:0}: Error finding container 2bddb93b6c7ea30c1a865c5e32f4b692a32343b06b6bc4d52d7ecb333e2fc355: Status 404 returned error can't find the container with id 2bddb93b6c7ea30c1a865c5e32f4b692a32343b06b6bc4d52d7ecb333e2fc355 Nov 03 05:55:17.712744 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.712711 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:17.712956 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:17.712942 4855 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 05:55:18.401358 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:18.401324 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqt2f" event={"ID":"08a92738-2cd0-4eed-9e83-c01baed110c2","Type":"ContainerStarted","Data":"521a10d57116723f0c08551916e2e67b76a90d6155bea22252d8ccf0ad8caeae"} Nov 03 05:55:18.401543 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:18.401417 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqt2f" event={"ID":"08a92738-2cd0-4eed-9e83-c01baed110c2","Type":"ContainerStarted","Data":"31bb44e55854c49250eb6d7818949a5df32d16b67a1c02fa83b2b3fbe8abdc12"} Nov 03 05:55:18.401543 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:18.401449 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqt2f" event={"ID":"08a92738-2cd0-4eed-9e83-c01baed110c2","Type":"ContainerStarted","Data":"2bddb93b6c7ea30c1a865c5e32f4b692a32343b06b6bc4d52d7ecb333e2fc355"} Nov 03 05:55:18.403167 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:18.403137 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" event={"ID":"99c39ff9-f070-4c29-8284-4f98d116faa4","Type":"ContainerStarted","Data":"00135296410df6627a13a2be081fec2d4c82e461eb13506c572bd8aadcd40906"} Nov 03 05:55:18.405174 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:18.405115 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" event={"ID":"56b77a5b-a55e-4849-9789-b9c5a3db4b02","Type":"ContainerStarted","Data":"d9a1aeeea93e92b314bff3ddfa976a74eaf0c10a161a6af051b3721947205f10"} Nov 03 05:55:18.527259 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:18.527232 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75h9" Nov 03 05:55:19.024788 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:19.024758 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-99n4t_36beafc8-4aee-448b-8dfd-64f89637aa9f/dns-node-resolver/0.log" Nov 03 05:55:20.224471 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:20.224444 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vvkcc_1eb02066-c20a-45b3-885a-0e9834b1c998/node-ca/0.log" Nov 03 05:55:20.417894 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:20.417854 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqt2f" event={"ID":"08a92738-2cd0-4eed-9e83-c01baed110c2","Type":"ContainerStarted","Data":"72aaa301acd1dd027042d5037da4cbe393218a84274ea8fa6bb4f26b959f3555"} Nov 03 05:55:20.419766 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:20.419741 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" event={"ID":"99c39ff9-f070-4c29-8284-4f98d116faa4","Type":"ContainerStarted","Data":"b02dff0bffffe3553a95facb76b2fcfa57bc3f1183b7286d004789d78b3ac2d2"} Nov 03 05:55:20.419900 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:20.419789 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d277j" event={"ID":"99c39ff9-f070-4c29-8284-4f98d116faa4","Type":"ContainerStarted","Data":"816db108656a0135e2020e270d5d23328b0e3f2e2bb5c0ded608543a04e59d53"} Nov 03 05:55:20.421225 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:20.421203 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vpktf" event={"ID":"56b77a5b-a55e-4849-9789-b9c5a3db4b02","Type":"ContainerStarted","Data":"d8e73d1ac06064c9a6270eef0cf8755af768ae4997810102fdae225c7d7db171"} Nov 03 05:55:20.824003 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:20.823974 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6j7pt_059a8188-2f82-4b0d-af82-5598a6e7b50e/serve-healthcheck-canary/0.log" Nov 03 05:55:21.141809 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:21.141726 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:21.141809 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:21.141773 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:21.141809 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:21.141790 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:21.141819 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:21.141846 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:21.141876 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.141883 4855 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.141903 4855 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c96d9b8b6-5hh78: secret "image-registry-tls" not found Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.141909 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle podName:76c09b47-0ab3-4212-924b-131d88e94ab3 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:29.141888832 +0000 UTC m=+16.950276098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle") pod "router-default-75b97bb597-swr58" (UID: "76c09b47-0ab3-4212-924b-131d88e94ab3") : configmap references non-existent config key: service-ca.crt Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.141923 4855 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.141941 4855 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67898f7b5f-gmmrp: secret "image-registry-tls" not found Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.141942 4855 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.141943 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls podName:74a0e103-b720-45e0-8f30-c54e9542b5cf nodeName:}" failed. No retries permitted until 2026-11-03 05:55:29.141929942 +0000 UTC m=+16.950317202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls") pod "image-registry-7c96d9b8b6-5hh78" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf") : secret "image-registry-tls" not found Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.141982 4855 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.142020 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs podName:58637e90-c0b9-44da-91e9-7e1075a2636e nodeName:}" failed. No retries permitted until 2026-11-03 05:55:29.142003559 +0000 UTC m=+16.950390808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs") pod "network-metrics-daemon-hbdpd" (UID: "58637e90-c0b9-44da-91e9-7e1075a2636e") : secret "metrics-daemon-secret" not found Nov 03 05:55:21.142047 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.142046 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls podName:f29d1b5f-b1e2-45e0-8b1f-da929d562b71 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:29.142033584 +0000 UTC m=+16.950420841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls") pod "image-registry-67898f7b5f-gmmrp" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71") : secret "image-registry-tls" not found Nov 03 05:55:21.142470 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:21.142064 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert podName:8ba1e446-7be2-4673-9a44-a6c6a5e1be2d nodeName:}" failed. No retries permitted until 2026-11-03 05:55:29.142057803 +0000 UTC m=+16.950445049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zknxx" (UID: "8ba1e446-7be2-4673-9a44-a6c6a5e1be2d") : secret "networking-console-plugin-cert" not found Nov 03 05:55:21.144161 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:21.144140 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd3a09f2-9d41-40f6-94f1-6cb00971d950-metrics-tls\") pod \"dns-default-g9htv\" (UID: \"bd3a09f2-9d41-40f6-94f1-6cb00971d950\") " pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:21.163988 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:21.163966 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:21.458896 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:21.458828 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g9htv"] Nov 03 05:55:21.461817 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:21.461792 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3a09f2_9d41_40f6_94f1_6cb00971d950.slice/crio-09a62e2849288038a2b9a823fb8c2b313df9bde3a3cfda5e21e0eba52939c766 WatchSource:0}: Error finding container 09a62e2849288038a2b9a823fb8c2b313df9bde3a3cfda5e21e0eba52939c766: Status 404 returned error can't find the container with id 09a62e2849288038a2b9a823fb8c2b313df9bde3a3cfda5e21e0eba52939c766 Nov 03 05:55:21.826186 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:21.826155 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7_80c48a6b-a95c-4c64-9d89-1ed9d0320908/kube-storage-version-migrator-operator/0.log" Nov 03 05:55:22.428537 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:22.428497 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g9htv" event={"ID":"bd3a09f2-9d41-40f6-94f1-6cb00971d950","Type":"ContainerStarted","Data":"09a62e2849288038a2b9a823fb8c2b313df9bde3a3cfda5e21e0eba52939c766"} Nov 03 05:55:22.754208 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:22.754119 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:22.754697 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:22.754588 4855 scope.go:122] "RemoveContainer" containerID="2278415587d3978e22ebfe07d162a1060d19e7a8bc4bad398af0e40dd5f9ab60" Nov 03 05:55:22.754850 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:22.754827 4855 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qwvxg_openshift-console-operator(2b099592-879f-4313-be96-60d39b44f8a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" podUID="2b099592-879f-4313-be96-60d39b44f8a8" Nov 03 05:55:23.009990 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:23.009958 4855 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:23.398729 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:23.398689 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:55:23.435731 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:23.435702 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g9htv" event={"ID":"bd3a09f2-9d41-40f6-94f1-6cb00971d950","Type":"ContainerStarted","Data":"81224bc5958e95079b24258414ce05cb993e1503dd089051c6637a4b33921a43"} Nov 03 05:55:23.435901 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:23.435753 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g9htv" event={"ID":"bd3a09f2-9d41-40f6-94f1-6cb00971d950","Type":"ContainerStarted","Data":"27229aa6b81535ab55d579e8f8e39ff97573f168a701f9d4ae9220f54e810abf"} Nov 03 05:55:23.435901 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:23.435846 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:23.436151 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:23.436137 4855 scope.go:122] "RemoveContainer" containerID="2278415587d3978e22ebfe07d162a1060d19e7a8bc4bad398af0e40dd5f9ab60" Nov 03 05:55:23.436323 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:23.436307 4855 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qwvxg_openshift-console-operator(2b099592-879f-4313-be96-60d39b44f8a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" podUID="2b099592-879f-4313-be96-60d39b44f8a8" Nov 03 05:55:24.439071 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:24.439037 4855 scope.go:122] "RemoveContainer" containerID="2278415587d3978e22ebfe07d162a1060d19e7a8bc4bad398af0e40dd5f9ab60" Nov 03 05:55:25.443758 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:25.443732 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 05:55:25.444237 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:25.443782 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" event={"ID":"2b099592-879f-4313-be96-60d39b44f8a8","Type":"ContainerStarted","Data":"74ae5b427f7a11101ad774b99a3783bdd3cef9cba54a61fe721b829008e76ed3"} Nov 03 05:55:25.444237 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:25.444059 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:25.448568 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:25.448546 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qwvxg" Nov 03 05:55:27.676282 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:27.676253 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sp2qh" Nov 03 05:55:28.882332 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:28.882301 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 05:55:29.217586 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.217488 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:29.217586 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.217535 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:29.217586 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.217556 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:29.217868 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.217597 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:29.217868 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.217624 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:29.218310 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.218284 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c09b47-0ab3-4212-924b-131d88e94ab3-service-ca-bundle\") pod \"router-default-75b97bb597-swr58\" (UID: \"76c09b47-0ab3-4212-924b-131d88e94ab3\") " pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:29.220025 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.219995 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ba1e446-7be2-4673-9a44-a6c6a5e1be2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zknxx\" (UID: \"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:29.220162 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.220142 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"image-registry-67898f7b5f-gmmrp\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:29.220349 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.220331 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"image-registry-7c96d9b8b6-5hh78\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:29.220384 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.220345 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58637e90-c0b9-44da-91e9-7e1075a2636e-metrics-certs\") pod \"network-metrics-daemon-hbdpd\" (UID: \"58637e90-c0b9-44da-91e9-7e1075a2636e\") " pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:29.252943 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.252913 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:29.255044 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.255024 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:29.256066 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.256051 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" Nov 03 05:55:29.258626 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.258607 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hbdpd" Nov 03 05:55:29.262328 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.262309 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:29.602846 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.602813 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75b97bb597-swr58"] Nov 03 05:55:29.605922 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:29.605893 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a0e103_b720_45e0_8f30_c54e9542b5cf.slice/crio-884392ca0d4781ceb2783a1e568f35347f4bd0e9a96cf99cbf7e5b4f377b4caf WatchSource:0}: Error finding container 884392ca0d4781ceb2783a1e568f35347f4bd0e9a96cf99cbf7e5b4f377b4caf: Status 404 returned error can't find the container with id 884392ca0d4781ceb2783a1e568f35347f4bd0e9a96cf99cbf7e5b4f377b4caf Nov 03 05:55:29.606447 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:29.606425 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c09b47_0ab3_4212_924b_131d88e94ab3.slice/crio-cd624f7836c317171460918d03a9c056c6cc41ef64ad58c6c15cab43a88ff312 WatchSource:0}: Error finding container cd624f7836c317171460918d03a9c056c6cc41ef64ad58c6c15cab43a88ff312: Status 404 returned error can't find the container with id cd624f7836c317171460918d03a9c056c6cc41ef64ad58c6c15cab43a88ff312 Nov 03 05:55:29.617274 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.617204 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c96d9b8b6-5hh78"] Nov 03 05:55:29.618013 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.617993 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zknxx"] Nov 03 05:55:29.618593 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.618572 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hbdpd"] Nov 03 05:55:29.620859 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:29.620836 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58637e90_c0b9_44da_91e9_7e1075a2636e.slice/crio-c0917a6b20de3e33de65a7d8a28a7f5b3688075a67fabe72738d49b133d739fd WatchSource:0}: Error finding container c0917a6b20de3e33de65a7d8a28a7f5b3688075a67fabe72738d49b133d739fd: Status 404 returned error can't find the container with id c0917a6b20de3e33de65a7d8a28a7f5b3688075a67fabe72738d49b133d739fd Nov 03 05:55:29.622416 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:29.621705 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba1e446_7be2_4673_9a44_a6c6a5e1be2d.slice/crio-1ea44b1cbf1e3d4f30ab7ad46a3fbb02881ed2a5e088a25bfecd4da65309aecb WatchSource:0}: Error finding container 1ea44b1cbf1e3d4f30ab7ad46a3fbb02881ed2a5e088a25bfecd4da65309aecb: Status 404 returned error can't find the container with id 1ea44b1cbf1e3d4f30ab7ad46a3fbb02881ed2a5e088a25bfecd4da65309aecb Nov 03 05:55:29.622623 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:29.622417 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29d1b5f_b1e2_45e0_8b1f_da929d562b71.slice/crio-d0a38b0fdf91342ad26d336176658e75a41128894bcdd2cbcf9fbbd822fab018 WatchSource:0}: Error finding container d0a38b0fdf91342ad26d336176658e75a41128894bcdd2cbcf9fbbd822fab018: Status 404 returned error can't find the container with id d0a38b0fdf91342ad26d336176658e75a41128894bcdd2cbcf9fbbd822fab018 Nov 03 05:55:29.630809 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:29.630789 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67898f7b5f-gmmrp"] Nov 03 05:55:30.462778 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.462741 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hbdpd" event={"ID":"58637e90-c0b9-44da-91e9-7e1075a2636e","Type":"ContainerStarted","Data":"c0917a6b20de3e33de65a7d8a28a7f5b3688075a67fabe72738d49b133d739fd"} Nov 03 05:55:30.464817 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.464784 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" event={"ID":"74a0e103-b720-45e0-8f30-c54e9542b5cf","Type":"ContainerStarted","Data":"e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964"} Nov 03 05:55:30.464958 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.464833 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" event={"ID":"74a0e103-b720-45e0-8f30-c54e9542b5cf","Type":"ContainerStarted","Data":"884392ca0d4781ceb2783a1e568f35347f4bd0e9a96cf99cbf7e5b4f377b4caf"} Nov 03 05:55:30.465090 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.465071 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:30.466678 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.466645 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" event={"ID":"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d","Type":"ContainerStarted","Data":"1ea44b1cbf1e3d4f30ab7ad46a3fbb02881ed2a5e088a25bfecd4da65309aecb"} Nov 03 05:55:30.468551 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.468523 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75b97bb597-swr58" event={"ID":"76c09b47-0ab3-4212-924b-131d88e94ab3","Type":"ContainerStarted","Data":"19297af81c67715d2502be0d9e7a1bdb88c012b2f14989face94935212de3bea"} Nov 03 05:55:30.468653 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.468573 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75b97bb597-swr58" event={"ID":"76c09b47-0ab3-4212-924b-131d88e94ab3","Type":"ContainerStarted","Data":"cd624f7836c317171460918d03a9c056c6cc41ef64ad58c6c15cab43a88ff312"} Nov 03 05:55:30.470349 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.470323 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" event={"ID":"f29d1b5f-b1e2-45e0-8b1f-da929d562b71","Type":"ContainerStarted","Data":"cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e"} Nov 03 05:55:30.470456 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.470367 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" event={"ID":"f29d1b5f-b1e2-45e0-8b1f-da929d562b71","Type":"ContainerStarted","Data":"d0a38b0fdf91342ad26d336176658e75a41128894bcdd2cbcf9fbbd822fab018"} Nov 03 05:55:30.470456 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.470450 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:30.578133 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:30.578102 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c96d9b8b6-5hh78"] Nov 03 05:55:31.254142 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:31.254107 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:31.256361 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:31.256337 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:31.475588 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:31.475539 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hbdpd" event={"ID":"58637e90-c0b9-44da-91e9-7e1075a2636e","Type":"ContainerStarted","Data":"3819ba71b655057445143e0f88167d7aa19817c73501c68a37faa2e35f806f41"} Nov 03 05:55:31.475989 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:31.475619 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hbdpd" event={"ID":"58637e90-c0b9-44da-91e9-7e1075a2636e","Type":"ContainerStarted","Data":"24cca4f2b057bf7457787147a09814301c8de338b3a27efd59a001713d4fd038"} Nov 03 05:55:31.477130 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:31.477095 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zknxx" event={"ID":"8ba1e446-7be2-4673-9a44-a6c6a5e1be2d","Type":"ContainerStarted","Data":"cad9c717467cbe42b6b06bfc3c1ec10c38996a4a1e7131a23550c307d0e4253a"} Nov 03 05:55:31.477565 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:31.477541 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:31.478494 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:31.478475 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75b97bb597-swr58" Nov 03 05:55:33.441214 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:33.441183 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g9htv" Nov 03 05:55:38.374285 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.374251 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-blr44"] Nov 03 05:55:38.405856 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.405824 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.422252 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.422226 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 05:55:38.422362 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.422265 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cbdsf\"" Nov 03 05:55:38.436207 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.436184 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 05:55:38.493255 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.493228 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-wtmp\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.493379 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.493259 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ggz\" (UniqueName: \"kubernetes.io/projected/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-kube-api-access-n9ggz\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.493379 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.493278 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-root\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.493379 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.493293 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-metrics-client-ca\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.493511 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.493382 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-textfile\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.493511 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.493449 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.493511 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.493471 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-tls\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.493511 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.493490 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-accelerators-collector-config\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.493632 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.493604 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-sys\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.510681 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.510656 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 05:55:38.510797 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.510705 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 05:55:38.594104 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594080 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-wtmp\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594224 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594107 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ggz\" (UniqueName: \"kubernetes.io/projected/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-kube-api-access-n9ggz\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594224 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594125 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-root\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594224 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594140 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-metrics-client-ca\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594224 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594163 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-textfile\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594224 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594181 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594224 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594204 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-tls\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594572 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594219 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-root\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594572 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594230 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-accelerators-collector-config\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594572 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594254 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-wtmp\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594572 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594323 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-sys\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594572 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594435 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-sys\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594830 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594628 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-textfile\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594830 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594806 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-metrics-client-ca\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.594830 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.594814 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-accelerators-collector-config\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.596713 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.596690 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.596771 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.596748 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-node-exporter-tls\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.618504 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.618482 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ggz\" (UniqueName: \"kubernetes.io/projected/2eaab29a-3c3a-4eb1-ae3b-ba66066b1144-kube-api-access-n9ggz\") pod \"node-exporter-blr44\" (UID: \"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144\") " pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.756734 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:38.756658 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-blr44" Nov 03 05:55:38.786060 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:55:38.786034 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eaab29a_3c3a_4eb1_ae3b_ba66066b1144.slice/crio-e36782d55478d1f5f367f7e963e3e33bfa4b56796c6eca4c297babdc11035569 WatchSource:0}: Error finding container e36782d55478d1f5f367f7e963e3e33bfa4b56796c6eca4c297babdc11035569: Status 404 returned error can't find the container with id e36782d55478d1f5f367f7e963e3e33bfa4b56796c6eca4c297babdc11035569 Nov 03 05:55:39.506309 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:39.506273 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blr44" event={"ID":"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144","Type":"ContainerStarted","Data":"e36782d55478d1f5f367f7e963e3e33bfa4b56796c6eca4c297babdc11035569"} Nov 03 05:55:40.511880 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:40.511799 4855 generic.go:356] "Generic (PLEG): container finished" podID="2eaab29a-3c3a-4eb1-ae3b-ba66066b1144" containerID="f1be5254a9269658e789f63fa2aeb43f32ae42dfb6aaee6e2ae49127c6cd8379" exitCode=0 Nov 03 05:55:40.511880 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:40.511849 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blr44" event={"ID":"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144","Type":"ContainerDied","Data":"f1be5254a9269658e789f63fa2aeb43f32ae42dfb6aaee6e2ae49127c6cd8379"} Nov 03 05:55:41.518060 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:41.518024 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blr44" event={"ID":"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144","Type":"ContainerStarted","Data":"4073ab76fb9db7bf5ee9aa8dfa607c40ade410d844abc3cbd630f2948c0517c2"} Nov 03 05:55:41.518575 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:41.518093 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blr44" event={"ID":"2eaab29a-3c3a-4eb1-ae3b-ba66066b1144","Type":"ContainerStarted","Data":"5369ef1b58343fb4029691411b8299d9324705a9530e2a6812f7038084498f49"} Nov 03 05:55:41.592283 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:41.592222 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-blr44" podStartSLOduration=2.1465712200000002 podStartE2EDuration="3.592209748s" totalImagesPullingTime="1.365734286s" totalInitContainerRuntime="79.904242ms" isStatefulPod=true podCreationTimestamp="2026-11-03 05:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:41.578089806 +0000 UTC m=+29.386477073" watchObservedRunningTime="2026-11-03 05:55:41.592209748 +0000 UTC m=+29.400597013" Nov 03 05:55:43.401680 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:43.401647 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cp5xp" Nov 03 05:55:51.481491 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:51.481463 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:55:52.486080 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:52.486050 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:53.964112 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:53.964032 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67898f7b5f-gmmrp"] Nov 03 05:55:57.506006 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.505936 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" podUID="74a0e103-b720-45e0-8f30-c54e9542b5cf" containerName="registry" containerID="cri-o://e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964" gracePeriod=30 Nov 03 05:55:57.752293 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.752268 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:57.861069 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.861038 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-kube-api-access-79lwr\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-kube-api-access-79lwr\") pod \"74a0e103-b720-45e0-8f30-c54e9542b5cf\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " Nov 03 05:55:57.861225 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.861079 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") pod \"74a0e103-b720-45e0-8f30-c54e9542b5cf\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " Nov 03 05:55:57.861225 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.861113 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a0e103-b720-45e0-8f30-c54e9542b5cf-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74a0e103-b720-45e0-8f30-c54e9542b5cf-ca-trust-extracted\") pod \"74a0e103-b720-45e0-8f30-c54e9542b5cf\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " Nov 03 05:55:57.861330 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.861253 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-installation-pull-secrets\") pod \"74a0e103-b720-45e0-8f30-c54e9542b5cf\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " Nov 03 05:55:57.861330 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.861315 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-certificates\") pod \"74a0e103-b720-45e0-8f30-c54e9542b5cf\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " Nov 03 05:55:57.861471 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.861378 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-image-registry-private-configuration\") pod \"74a0e103-b720-45e0-8f30-c54e9542b5cf\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " Nov 03 05:55:57.861540 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.861522 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-bound-sa-token\") pod \"74a0e103-b720-45e0-8f30-c54e9542b5cf\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " Nov 03 05:55:57.861636 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.861612 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-trusted-ca\") pod \"74a0e103-b720-45e0-8f30-c54e9542b5cf\" (UID: \"74a0e103-b720-45e0-8f30-c54e9542b5cf\") " Nov 03 05:55:57.861920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.861774 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-certificates" pod "74a0e103-b720-45e0-8f30-c54e9542b5cf" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 05:55:57.861920 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.861882 4855 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-certificates\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:55:57.862246 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.862223 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-trusted-ca" pod "74a0e103-b720-45e0-8f30-c54e9542b5cf" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 05:55:57.863877 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.863852 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls" pod "74a0e103-b720-45e0-8f30-c54e9542b5cf" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 05:55:57.863979 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.863961 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-installation-pull-secrets" pod "74a0e103-b720-45e0-8f30-c54e9542b5cf" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 05:55:57.864058 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.864034 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-kube-api-access-79lwr" pod "74a0e103-b720-45e0-8f30-c54e9542b5cf" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf"). InnerVolumeSpecName "kube-api-access-79lwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 05:55:57.864117 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.864098 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-bound-sa-token" pod "74a0e103-b720-45e0-8f30-c54e9542b5cf" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 05:55:57.864153 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.864137 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-image-registry-private-configuration" pod "74a0e103-b720-45e0-8f30-c54e9542b5cf" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 05:55:57.868458 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.868436 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a0e103-b720-45e0-8f30-c54e9542b5cf-ca-trust-extracted" pod "74a0e103-b720-45e0-8f30-c54e9542b5cf" (UID: "74a0e103-b720-45e0-8f30-c54e9542b5cf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 05:55:57.962646 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.962613 4855 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-bound-sa-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:55:57.962646 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.962647 4855 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74a0e103-b720-45e0-8f30-c54e9542b5cf-trusted-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:55:57.962858 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.962661 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79lwr\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-kube-api-access-79lwr\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:55:57.962858 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.962672 4855 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74a0e103-b720-45e0-8f30-c54e9542b5cf-registry-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:55:57.962858 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.962681 4855 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74a0e103-b720-45e0-8f30-c54e9542b5cf-ca-trust-extracted\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:55:57.962858 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.962691 4855 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-installation-pull-secrets\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:55:57.962858 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:57.962701 4855 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74a0e103-b720-45e0-8f30-c54e9542b5cf-image-registry-private-configuration\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:55:58.581806 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:58.581769 4855 generic.go:356] "Generic (PLEG): container finished" podID="74a0e103-b720-45e0-8f30-c54e9542b5cf" containerID="e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964" exitCode=0 Nov 03 05:55:58.582278 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:58.581826 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" Nov 03 05:55:58.582278 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:58.581864 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" event={"ID":"74a0e103-b720-45e0-8f30-c54e9542b5cf","Type":"ContainerDied","Data":"e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964"} Nov 03 05:55:58.582278 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:58.581928 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c96d9b8b6-5hh78" event={"ID":"74a0e103-b720-45e0-8f30-c54e9542b5cf","Type":"ContainerDied","Data":"884392ca0d4781ceb2783a1e568f35347f4bd0e9a96cf99cbf7e5b4f377b4caf"} Nov 03 05:55:58.582278 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:58.581961 4855 scope.go:122] "RemoveContainer" containerID="e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964" Nov 03 05:55:58.589897 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:58.589880 4855 scope.go:122] "RemoveContainer" containerID="e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964" Nov 03 05:55:58.590163 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:58.590144 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964\": container with ID starting with e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964 not found: ID does not exist" containerID="e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964" Nov 03 05:55:58.590203 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:58.590173 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964"} err="failed to get container status \"e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964\": rpc error: code = NotFound desc = could not find container \"e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964\": container with ID starting with e664e9be05c41cb65c3dfd420eabc1e777f66e1e80c15401e54ccf67c67bb964 not found: ID does not exist" Nov 03 05:55:58.651932 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:58.651906 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c96d9b8b6-5hh78"] Nov 03 05:55:58.665681 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:55:58.665654 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c96d9b8b6-5hh78"] Nov 03 05:55:58.884049 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:55:58.883967 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 05:56:00.333599 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:00.331703 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a0e103-b720-45e0-8f30-c54e9542b5cf" path="/var/lib/kubelet/pods/74a0e103-b720-45e0-8f30-c54e9542b5cf/volumes" Nov 03 05:56:18.983996 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:18.983931 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" podUID="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" containerName="registry" containerID="cri-o://cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e" gracePeriod=30 Nov 03 05:56:19.217348 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.217325 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:56:19.324458 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.324428 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-ca-trust-extracted\") pod \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " Nov 03 05:56:19.324609 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.324468 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-kube-api-access-gl4ch\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-kube-api-access-gl4ch\") pod \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " Nov 03 05:56:19.324609 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.324488 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-certificates\") pod \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " Nov 03 05:56:19.324609 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.324509 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-image-registry-private-configuration\") pod \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " Nov 03 05:56:19.324609 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.324539 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-trusted-ca\") pod \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " Nov 03 05:56:19.324609 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.324584 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-installation-pull-secrets\") pod \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " Nov 03 05:56:19.324882 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.324614 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-bound-sa-token\") pod \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " Nov 03 05:56:19.324942 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.324890 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") pod \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\" (UID: \"f29d1b5f-b1e2-45e0-8b1f-da929d562b71\") " Nov 03 05:56:19.325007 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.324955 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-certificates" pod "f29d1b5f-b1e2-45e0-8b1f-da929d562b71" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 05:56:19.325058 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.325017 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-trusted-ca" pod "f29d1b5f-b1e2-45e0-8b1f-da929d562b71" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 05:56:19.325201 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.325175 4855 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-certificates\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:56:19.325201 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.325203 4855 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-trusted-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:56:19.327064 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.327010 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls" pod "f29d1b5f-b1e2-45e0-8b1f-da929d562b71" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 05:56:19.327227 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.327199 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-installation-pull-secrets" pod "f29d1b5f-b1e2-45e0-8b1f-da929d562b71" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 05:56:19.327362 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.327343 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-image-registry-private-configuration" pod "f29d1b5f-b1e2-45e0-8b1f-da929d562b71" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 05:56:19.327449 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.327416 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-kube-api-access-gl4ch" pod "f29d1b5f-b1e2-45e0-8b1f-da929d562b71" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71"). InnerVolumeSpecName "kube-api-access-gl4ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 05:56:19.327496 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.327451 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-bound-sa-token" pod "f29d1b5f-b1e2-45e0-8b1f-da929d562b71" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 05:56:19.331051 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.331028 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-ca-trust-extracted" pod "f29d1b5f-b1e2-45e0-8b1f-da929d562b71" (UID: "f29d1b5f-b1e2-45e0-8b1f-da929d562b71"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 05:56:19.425630 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.425604 4855 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-installation-pull-secrets\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:56:19.425630 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.425627 4855 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-bound-sa-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:56:19.425630 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.425637 4855 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-registry-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:56:19.425832 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.425645 4855 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-ca-trust-extracted\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:56:19.425832 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.425655 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl4ch\" (UniqueName: \"kubernetes.io/projected/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-kube-api-access-gl4ch\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:56:19.425832 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.425665 4855 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f29d1b5f-b1e2-45e0-8b1f-da929d562b71-image-registry-private-configuration\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 05:56:19.660084 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.660001 4855 generic.go:356] "Generic (PLEG): container finished" podID="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" containerID="cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e" exitCode=0 Nov 03 05:56:19.660084 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.660049 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" Nov 03 05:56:19.660318 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.660097 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" event={"ID":"f29d1b5f-b1e2-45e0-8b1f-da929d562b71","Type":"ContainerDied","Data":"cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e"} Nov 03 05:56:19.660318 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.660150 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67898f7b5f-gmmrp" event={"ID":"f29d1b5f-b1e2-45e0-8b1f-da929d562b71","Type":"ContainerDied","Data":"d0a38b0fdf91342ad26d336176658e75a41128894bcdd2cbcf9fbbd822fab018"} Nov 03 05:56:19.660318 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.660176 4855 scope.go:122] "RemoveContainer" containerID="cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e" Nov 03 05:56:19.668809 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.668787 4855 scope.go:122] "RemoveContainer" containerID="cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e" Nov 03 05:56:19.669055 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:56:19.669038 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e\": container with ID starting with cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e not found: ID does not exist" containerID="cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e" Nov 03 05:56:19.669122 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.669062 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e"} err="failed to get container status \"cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e\": rpc error: code = NotFound desc = could not find container \"cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e\": container with ID starting with cac7634b50d4696dcdd895b9c28b8d963128225fcdda385893e44a0f78ac174e not found: ID does not exist" Nov 03 05:56:19.715070 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.715049 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67898f7b5f-gmmrp"] Nov 03 05:56:19.719159 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:19.719142 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67898f7b5f-gmmrp"] Nov 03 05:56:20.330745 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:20.330716 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" path="/var/lib/kubelet/pods/f29d1b5f-b1e2-45e0-8b1f-da929d562b71/volumes" Nov 03 05:56:28.694514 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:28.694472 4855 generic.go:356] "Generic (PLEG): container finished" podID="80c48a6b-a95c-4c64-9d89-1ed9d0320908" containerID="f2cc64f408500b0480ba2548f6d6b1af6098e971ae45f6d13bc561078c2065b0" exitCode=0 Nov 03 05:56:28.694907 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:28.694544 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" event={"ID":"80c48a6b-a95c-4c64-9d89-1ed9d0320908","Type":"ContainerDied","Data":"f2cc64f408500b0480ba2548f6d6b1af6098e971ae45f6d13bc561078c2065b0"} Nov 03 05:56:28.694907 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:28.694841 4855 scope.go:122] "RemoveContainer" containerID="f2cc64f408500b0480ba2548f6d6b1af6098e971ae45f6d13bc561078c2065b0" Nov 03 05:56:28.882001 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:56:28.881968 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 05:56:29.699582 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:29.699546 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lg4f7" event={"ID":"80c48a6b-a95c-4c64-9d89-1ed9d0320908","Type":"ContainerStarted","Data":"9ffc25f91f39b2789d7b203520424575b44d0683c32b82e5843c819ec6ac5a52"} Nov 03 05:56:33.714751 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:33.714712 4855 generic.go:356] "Generic (PLEG): container finished" podID="315726a9-3009-40f6-bf65-6818acd95046" containerID="5d615acf6c9415b0697ea23a73d415cacb33a017e38ad366737b36c5ce63b5f0" exitCode=0 Nov 03 05:56:33.714751 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:33.714753 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-86pnb" event={"ID":"315726a9-3009-40f6-bf65-6818acd95046","Type":"ContainerDied","Data":"5d615acf6c9415b0697ea23a73d415cacb33a017e38ad366737b36c5ce63b5f0"} Nov 03 05:56:33.715304 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:33.715141 4855 scope.go:122] "RemoveContainer" containerID="5d615acf6c9415b0697ea23a73d415cacb33a017e38ad366737b36c5ce63b5f0" Nov 03 05:56:34.720594 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:34.720554 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-86pnb" event={"ID":"315726a9-3009-40f6-bf65-6818acd95046","Type":"ContainerStarted","Data":"a09eeec0860d06c0acaafc92468761301f93ab526b8ec372545740c10c229f2f"} Nov 03 05:56:43.756996 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:43.756956 4855 generic.go:356] "Generic (PLEG): container finished" podID="49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f" containerID="cadfecc8bda7de97d34adf474edaa53fca1cb15309a5415eb5e1a87ddda409e4" exitCode=0 Nov 03 05:56:43.757506 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:43.757029 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" event={"ID":"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f","Type":"ContainerDied","Data":"cadfecc8bda7de97d34adf474edaa53fca1cb15309a5415eb5e1a87ddda409e4"} Nov 03 05:56:43.757506 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:43.757411 4855 scope.go:122] "RemoveContainer" containerID="cadfecc8bda7de97d34adf474edaa53fca1cb15309a5415eb5e1a87ddda409e4" Nov 03 05:56:44.762314 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:56:44.762282 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g27mq" event={"ID":"49f4a643-0b2d-45ca-9ab7-3ad2ae287a8f","Type":"ContainerStarted","Data":"54b6b367de414fbbf81279a9f49cdd4701419fd0f77c2ce4f652f85796dd0d5d"} Nov 03 05:56:58.882690 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:56:58.882642 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 05:57:28.881873 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:57:28.881793 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 05:57:58.882911 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:57:58.882873 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 05:58:16.217007 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.216968 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k"] Nov 03 05:58:16.217445 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.217269 4855 cpu_manager.go:414] "containerMap: removing container" podUID="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" containerName="registry" Nov 03 05:58:16.217445 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.217280 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" containerName="registry" Nov 03 05:58:16.217445 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.217302 4855 cpu_manager.go:414] "containerMap: removing container" podUID="74a0e103-b720-45e0-8f30-c54e9542b5cf" containerName="registry" Nov 03 05:58:16.217445 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.217308 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a0e103-b720-45e0-8f30-c54e9542b5cf" containerName="registry" Nov 03 05:58:16.217445 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.217370 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29d1b5f-b1e2-45e0-8b1f-da929d562b71" containerName="registry" Nov 03 05:58:16.217445 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.217382 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a0e103-b720-45e0-8f30-c54e9542b5cf" containerName="registry" Nov 03 05:58:16.220231 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.220214 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.237015 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.236995 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 05:58:16.237131 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.237001 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 05:58:16.237131 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.237066 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 05:58:16.237131 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.237118 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 05:58:16.237283 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.237205 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mrr85\"" Nov 03 05:58:16.251916 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.251891 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k"] Nov 03 05:58:16.324585 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.324551 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.324585 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.324590 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6be883-9666-4f6d-9b4b-468972aa036a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.324795 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.324618 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.324795 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.324748 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.324795 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.324776 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z9k6\" (UniqueName: \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-kube-api-access-7z9k6\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.324888 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.324807 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.324888 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.324826 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.425850 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.425822 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.425850 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.425853 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6be883-9666-4f6d-9b4b-468972aa036a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.426083 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.425874 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.426083 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.425899 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.426083 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.425933 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z9k6\" (UniqueName: \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-kube-api-access-7z9k6\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.426083 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.425966 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.426083 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.425995 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.426630 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.426598 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.428259 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.428234 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.428352 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.428239 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.428431 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.428381 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6be883-9666-4f6d-9b4b-468972aa036a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.428544 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.428526 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.450706 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.450683 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.450876 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.450858 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z9k6\" (UniqueName: \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-kube-api-access-7z9k6\") pod \"istiod-openshift-gateway-7876bfcb74-vjn6k\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.531144 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.531062 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:16.842565 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:16.842532 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k"] Nov 03 05:58:16.846157 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:58:16.846132 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6be883_9666_4f6d_9b4b_468972aa036a.slice/crio-f1035839ec8c40834942c18695284d29f5d6022244f645092f1e865dcb577b71 WatchSource:0}: Error finding container f1035839ec8c40834942c18695284d29f5d6022244f645092f1e865dcb577b71: Status 404 returned error can't find the container with id f1035839ec8c40834942c18695284d29f5d6022244f645092f1e865dcb577b71 Nov 03 05:58:17.073411 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:17.073362 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" event={"ID":"df6be883-9666-4f6d-9b4b-468972aa036a","Type":"ContainerStarted","Data":"f1035839ec8c40834942c18695284d29f5d6022244f645092f1e865dcb577b71"} Nov 03 05:58:23.153571 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:23.153529 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 05:58:23.153829 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:23.153610 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 05:58:24.101148 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:24.101110 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" event={"ID":"df6be883-9666-4f6d-9b4b-468972aa036a","Type":"ContainerStarted","Data":"d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90"} Nov 03 05:58:24.101341 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:24.101287 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:24.102909 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:24.102884 4855 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-vjn6k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 05:58:24.103056 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:24.102930 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" podUID="df6be883-9666-4f6d-9b4b-468972aa036a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 05:58:24.168743 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:24.168688 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" podStartSLOduration=1.864015571 podStartE2EDuration="8.168674071s" totalImagesPullingTime="6.3046585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:58:24.164257834 +0000 UTC m=+191.972645103" watchObservedRunningTime="2026-11-03 05:58:24.168674071 +0000 UTC m=+191.977061340" Nov 03 05:58:25.106036 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:58:25.106009 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 05:58:28.882535 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:58:28.882500 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 05:58:58.882870 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:58:58.882798 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 05:59:06.172377 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.172342 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp"] Nov 03 05:59:06.175664 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.175647 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.224336 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.224313 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 05:59:06.224336 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.224329 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 05:59:06.224533 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.224433 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 05:59:06.224533 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.224444 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 05:59:06.224533 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.224464 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5996c\"" Nov 03 05:59:06.224533 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.224495 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 05:59:06.232031 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.232005 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp"] Nov 03 05:59:06.307947 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.307917 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae748b82-61ea-4921-902f-76396e924505-tmp\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.308119 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.307967 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae748b82-61ea-4921-902f-76396e924505-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.308119 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.308047 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae748b82-61ea-4921-902f-76396e924505-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.308119 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.308093 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdbt\" (UniqueName: \"kubernetes.io/projected/ae748b82-61ea-4921-902f-76396e924505-kube-api-access-zkdbt\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.308295 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.308182 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae748b82-61ea-4921-902f-76396e924505-manifests\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.308295 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.308243 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae748b82-61ea-4921-902f-76396e924505-metrics-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.409321 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.409291 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae748b82-61ea-4921-902f-76396e924505-tmp\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.409492 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.409438 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae748b82-61ea-4921-902f-76396e924505-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.409492 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.409478 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae748b82-61ea-4921-902f-76396e924505-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.409583 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.409513 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdbt\" (UniqueName: \"kubernetes.io/projected/ae748b82-61ea-4921-902f-76396e924505-kube-api-access-zkdbt\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.409583 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.409532 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae748b82-61ea-4921-902f-76396e924505-manifests\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.409685 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.409587 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae748b82-61ea-4921-902f-76396e924505-metrics-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.409924 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.409889 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae748b82-61ea-4921-902f-76396e924505-manifests\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.411493 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.411460 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae748b82-61ea-4921-902f-76396e924505-tmp\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.411865 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.411849 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae748b82-61ea-4921-902f-76396e924505-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.411906 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.411862 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae748b82-61ea-4921-902f-76396e924505-metrics-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.411940 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.411931 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae748b82-61ea-4921-902f-76396e924505-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.434776 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.434716 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdbt\" (UniqueName: \"kubernetes.io/projected/ae748b82-61ea-4921-902f-76396e924505-kube-api-access-zkdbt\") pod \"opendatahub-operator-controller-manager-54b885cf97-dlsgp\" (UID: \"ae748b82-61ea-4921-902f-76396e924505\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.490861 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.490827 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:06.850020 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:06.849955 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp"] Nov 03 05:59:06.852344 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:59:06.852314 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae748b82_61ea_4921_902f_76396e924505.slice/crio-e5f646979318ef9aade78459cd0edf4f6819175306cea5f57d3d5b670d330a09 WatchSource:0}: Error finding container e5f646979318ef9aade78459cd0edf4f6819175306cea5f57d3d5b670d330a09: Status 404 returned error can't find the container with id e5f646979318ef9aade78459cd0edf4f6819175306cea5f57d3d5b670d330a09 Nov 03 05:59:07.255161 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:07.255087 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" event={"ID":"ae748b82-61ea-4921-902f-76396e924505","Type":"ContainerStarted","Data":"e5f646979318ef9aade78459cd0edf4f6819175306cea5f57d3d5b670d330a09"} Nov 03 05:59:10.269016 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:10.268984 4855 generic.go:356] "Generic (PLEG): container finished" podID="ae748b82-61ea-4921-902f-76396e924505" containerID="406f43ab409611c1c3229dd042a7a83f9e1d5e03b8a83836a7c9ae37a51cd6a6" exitCode=0 Nov 03 05:59:10.269016 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:10.269020 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" event={"ID":"ae748b82-61ea-4921-902f-76396e924505","Type":"ContainerDied","Data":"406f43ab409611c1c3229dd042a7a83f9e1d5e03b8a83836a7c9ae37a51cd6a6"} Nov 03 05:59:11.275878 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:11.275842 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" event={"ID":"ae748b82-61ea-4921-902f-76396e924505","Type":"ContainerStarted","Data":"990f02aa972662244b221a24fc493f8e96d6ccc3aac72a467b6a0510598a7bb4"} Nov 03 05:59:11.276323 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:11.275980 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:11.410221 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:11.410167 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" podStartSLOduration=2.004199379 podStartE2EDuration="5.410150373s" totalImagesPullingTime="2.971754278s" totalInitContainerRuntime="434.196716ms" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:11.395892998 +0000 UTC m=+239.204280267" watchObservedRunningTime="2026-11-03 05:59:11.410150373 +0000 UTC m=+239.218537641" Nov 03 05:59:22.282184 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:22.282152 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-dlsgp" Nov 03 05:59:28.883077 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:59:28.883039 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 05:59:40.350495 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.350462 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh"] Nov 03 05:59:40.353750 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.353736 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:40.373225 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.373198 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-tls-certs\") pod \"kube-auth-proxy-d95cd9746-lvxsh\" (UID: \"f7fa46c9-0a19-4980-b190-06d98b2aaf8e\") " pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:40.373413 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.373236 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfc4\" (UniqueName: \"kubernetes.io/projected/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-kube-api-access-vrfc4\") pod \"kube-auth-proxy-d95cd9746-lvxsh\" (UID: \"f7fa46c9-0a19-4980-b190-06d98b2aaf8e\") " pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:40.373413 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.373329 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-tmp\") pod \"kube-auth-proxy-d95cd9746-lvxsh\" (UID: \"f7fa46c9-0a19-4980-b190-06d98b2aaf8e\") " pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:40.374890 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.374867 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 05:59:40.374991 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.374921 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 05:59:40.375181 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.375164 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6dhm6\"" Nov 03 05:59:40.391082 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.391051 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh"] Nov 03 05:59:40.474252 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.474220 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-tls-certs\") pod \"kube-auth-proxy-d95cd9746-lvxsh\" (UID: \"f7fa46c9-0a19-4980-b190-06d98b2aaf8e\") " pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:40.474252 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.474258 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfc4\" (UniqueName: \"kubernetes.io/projected/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-kube-api-access-vrfc4\") pod \"kube-auth-proxy-d95cd9746-lvxsh\" (UID: \"f7fa46c9-0a19-4980-b190-06d98b2aaf8e\") " pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:40.474524 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.474295 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-tmp\") pod \"kube-auth-proxy-d95cd9746-lvxsh\" (UID: \"f7fa46c9-0a19-4980-b190-06d98b2aaf8e\") " pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:40.474524 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:59:40.474372 4855 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Nov 03 05:59:40.474524 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:59:40.474467 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-tls-certs podName:f7fa46c9-0a19-4980-b190-06d98b2aaf8e nodeName:}" failed. No retries permitted until 2026-11-03 05:59:40.974446936 +0000 UTC m=+268.782834197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-tls-certs") pod "kube-auth-proxy-d95cd9746-lvxsh" (UID: "f7fa46c9-0a19-4980-b190-06d98b2aaf8e") : secret "kube-auth-proxy-tls" not found Nov 03 05:59:40.476488 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.476468 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-tmp\") pod \"kube-auth-proxy-d95cd9746-lvxsh\" (UID: \"f7fa46c9-0a19-4980-b190-06d98b2aaf8e\") " pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:40.499572 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.499548 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfc4\" (UniqueName: \"kubernetes.io/projected/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-kube-api-access-vrfc4\") pod \"kube-auth-proxy-d95cd9746-lvxsh\" (UID: \"f7fa46c9-0a19-4980-b190-06d98b2aaf8e\") " pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:40.978040 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.978004 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-tls-certs\") pod \"kube-auth-proxy-d95cd9746-lvxsh\" (UID: \"f7fa46c9-0a19-4980-b190-06d98b2aaf8e\") " pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:40.980516 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:40.980486 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7fa46c9-0a19-4980-b190-06d98b2aaf8e-tls-certs\") pod \"kube-auth-proxy-d95cd9746-lvxsh\" (UID: \"f7fa46c9-0a19-4980-b190-06d98b2aaf8e\") " pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:41.266049 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:41.265970 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" Nov 03 05:59:41.567963 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:41.567941 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh"] Nov 03 05:59:41.570165 ip-10-0-141-67 kubenswrapper[4855]: W1103 05:59:41.570133 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fa46c9_0a19_4980_b190_06d98b2aaf8e.slice/crio-6caa879081316b49640b32be36c53edb568fcae340e48ad53d1122439db0e479 WatchSource:0}: Error finding container 6caa879081316b49640b32be36c53edb568fcae340e48ad53d1122439db0e479: Status 404 returned error can't find the container with id 6caa879081316b49640b32be36c53edb568fcae340e48ad53d1122439db0e479 Nov 03 05:59:42.394697 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:42.394651 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" event={"ID":"f7fa46c9-0a19-4980-b190-06d98b2aaf8e","Type":"ContainerStarted","Data":"6caa879081316b49640b32be36c53edb568fcae340e48ad53d1122439db0e479"} Nov 03 05:59:44.405898 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:44.405859 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" event={"ID":"f7fa46c9-0a19-4980-b190-06d98b2aaf8e","Type":"ContainerStarted","Data":"af2ff8280f09b10e1b4bb43daf07e88449dd322ba92c4cf0560f56c1eca2ee61"} Nov 03 05:59:44.449987 ip-10-0-141-67 kubenswrapper[4855]: I1103 05:59:44.449941 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d95cd9746-lvxsh" podStartSLOduration=2.305068104 podStartE2EDuration="4.449926279s" totalImagesPullingTime="2.144858175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:44.449190133 +0000 UTC m=+272.257577402" watchObservedRunningTime="2026-11-03 05:59:44.449926279 +0000 UTC m=+272.258313541" Nov 03 05:59:58.883281 ip-10-0-141-67 kubenswrapper[4855]: E1103 05:59:58.883244 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:00:12.350577 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:12.350545 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:00:12.351460 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:12.351434 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:00:20.758635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.758597 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:00:20.764929 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.764908 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" Nov 03 06:00:20.782464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.782439 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 06:00:20.795913 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.795879 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 06:00:20.795913 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.795890 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 06:00:20.796079 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.795920 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qtbtg\"" Nov 03 06:00:20.797074 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.797055 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:00:20.848311 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.848283 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:00:20.890441 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.890415 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0270da45-d870-4752-aef3-85f92cebb57e-config-file\") pod \"limitador-limitador-7c9b87d574-qfq2f\" (UID: \"0270da45-d870-4752-aef3-85f92cebb57e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" Nov 03 06:00:20.890532 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.890475 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzd6\" (UniqueName: \"kubernetes.io/projected/0270da45-d870-4752-aef3-85f92cebb57e-kube-api-access-mhzd6\") pod \"limitador-limitador-7c9b87d574-qfq2f\" (UID: \"0270da45-d870-4752-aef3-85f92cebb57e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" Nov 03 06:00:20.991797 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.991753 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzd6\" (UniqueName: \"kubernetes.io/projected/0270da45-d870-4752-aef3-85f92cebb57e-kube-api-access-mhzd6\") pod \"limitador-limitador-7c9b87d574-qfq2f\" (UID: \"0270da45-d870-4752-aef3-85f92cebb57e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" Nov 03 06:00:20.991977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.991846 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0270da45-d870-4752-aef3-85f92cebb57e-config-file\") pod \"limitador-limitador-7c9b87d574-qfq2f\" (UID: \"0270da45-d870-4752-aef3-85f92cebb57e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" Nov 03 06:00:20.992521 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:20.992502 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0270da45-d870-4752-aef3-85f92cebb57e-config-file\") pod \"limitador-limitador-7c9b87d574-qfq2f\" (UID: \"0270da45-d870-4752-aef3-85f92cebb57e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" Nov 03 06:00:21.016842 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:21.016789 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzd6\" (UniqueName: \"kubernetes.io/projected/0270da45-d870-4752-aef3-85f92cebb57e-kube-api-access-mhzd6\") pod \"limitador-limitador-7c9b87d574-qfq2f\" (UID: \"0270da45-d870-4752-aef3-85f92cebb57e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" Nov 03 06:00:21.077815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:21.077786 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" Nov 03 06:00:21.376187 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:21.376147 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:00:21.379327 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:00:21.379301 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0270da45_d870_4752_aef3_85f92cebb57e.slice/crio-b9a18bdfdc9f86a564f73d6634adc4fc2444494eec755bbf1ad2c4c0f35c6d94 WatchSource:0}: Error finding container b9a18bdfdc9f86a564f73d6634adc4fc2444494eec755bbf1ad2c4c0f35c6d94: Status 404 returned error can't find the container with id b9a18bdfdc9f86a564f73d6634adc4fc2444494eec755bbf1ad2c4c0f35c6d94 Nov 03 06:00:21.381786 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:21.381767 4855 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 06:00:21.553771 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:21.553727 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" event={"ID":"0270da45-d870-4752-aef3-85f92cebb57e","Type":"ContainerStarted","Data":"b9a18bdfdc9f86a564f73d6634adc4fc2444494eec755bbf1ad2c4c0f35c6d94"} Nov 03 06:00:24.567433 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:24.567402 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" event={"ID":"0270da45-d870-4752-aef3-85f92cebb57e","Type":"ContainerStarted","Data":"0a28ab7a9b6460d6d0de992f841d20dcc5eb9eb46f9a3c5c0e1a481072d4e4ed"} Nov 03 06:00:24.567838 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:24.567485 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" Nov 03 06:00:24.611738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:24.611682 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" podStartSLOduration=1.7088503400000001 podStartE2EDuration="4.611669168s" totalImagesPullingTime="2.902818828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:00:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:00:24.61113019 +0000 UTC m=+312.419517460" watchObservedRunningTime="2026-11-03 06:00:24.611669168 +0000 UTC m=+312.420056433" Nov 03 06:00:28.882403 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:00:28.882348 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:00:35.572791 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:35.572765 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qfq2f" Nov 03 06:00:43.432057 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.432023 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg"] Nov 03 06:00:43.439793 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.439771 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.475770 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.475733 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg"] Nov 03 06:00:43.595741 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.595702 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.595741 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.595737 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.596004 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.595761 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5cae47d3-8f00-409f-8a46-6d982382e33b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.596004 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.595896 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.596004 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.595959 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.596126 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.596012 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5r7w\" (UniqueName: \"kubernetes.io/projected/5cae47d3-8f00-409f-8a46-6d982382e33b-kube-api-access-v5r7w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.596126 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.596063 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cae47d3-8f00-409f-8a46-6d982382e33b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.697173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.697093 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cae47d3-8f00-409f-8a46-6d982382e33b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.697173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.697168 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.697423 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.697192 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.697423 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.697229 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5cae47d3-8f00-409f-8a46-6d982382e33b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.697423 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.697259 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.697423 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.697299 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.697423 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.697323 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5r7w\" (UniqueName: \"kubernetes.io/projected/5cae47d3-8f00-409f-8a46-6d982382e33b-kube-api-access-v5r7w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.698199 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.698168 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.699722 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.699693 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cae47d3-8f00-409f-8a46-6d982382e33b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.699818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.699727 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.699920 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.699897 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5cae47d3-8f00-409f-8a46-6d982382e33b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.699955 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.699901 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.723687 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.723661 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5r7w\" (UniqueName: \"kubernetes.io/projected/5cae47d3-8f00-409f-8a46-6d982382e33b-kube-api-access-v5r7w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.743720 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.743691 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cae47d3-8f00-409f-8a46-6d982382e33b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r2pjg\" (UID: \"5cae47d3-8f00-409f-8a46-6d982382e33b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:43.751584 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:43.751564 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:44.069200 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:44.069172 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg"] Nov 03 06:00:44.071919 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:00:44.071878 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cae47d3_8f00_409f_8a46_6d982382e33b.slice/crio-3eb7fdb6fd8c4af076208b513005d236842c919025a71510a6ac8a1e10f62255 WatchSource:0}: Error finding container 3eb7fdb6fd8c4af076208b513005d236842c919025a71510a6ac8a1e10f62255: Status 404 returned error can't find the container with id 3eb7fdb6fd8c4af076208b513005d236842c919025a71510a6ac8a1e10f62255 Nov 03 06:00:44.074353 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:44.074320 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:00:44.074438 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:44.074378 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:00:44.647728 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:44.647689 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" event={"ID":"5cae47d3-8f00-409f-8a46-6d982382e33b","Type":"ContainerStarted","Data":"55182e14c2e7ea4187f57da8a5a7636556e4b6c1525d427fc2ae9b3c4e63f45f"} Nov 03 06:00:44.648179 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:44.647741 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" event={"ID":"5cae47d3-8f00-409f-8a46-6d982382e33b","Type":"ContainerStarted","Data":"3eb7fdb6fd8c4af076208b513005d236842c919025a71510a6ac8a1e10f62255"} Nov 03 06:00:44.648179 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:44.647950 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:44.697529 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:44.697475 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" podStartSLOduration=1.697457876 podStartE2EDuration="1.697457876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:00:44.696207702 +0000 UTC m=+332.504594971" watchObservedRunningTime="2026-11-03 06:00:44.697457876 +0000 UTC m=+332.505845143" Nov 03 06:00:45.653893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:45.653864 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r2pjg" Nov 03 06:00:45.750243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:45.750210 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k"] Nov 03 06:00:45.750469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:45.750431 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" podUID="df6be883-9666-4f6d-9b4b-468972aa036a" containerName="discovery" containerID="cri-o://d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90" gracePeriod=30 Nov 03 06:00:45.997980 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:45.997956 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 06:00:46.018639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.018611 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-ca-configmap\") pod \"df6be883-9666-4f6d-9b4b-468972aa036a\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " Nov 03 06:00:46.018775 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.018652 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6be883-9666-4f6d-9b4b-468972aa036a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6be883-9666-4f6d-9b4b-468972aa036a-local-certs\") pod \"df6be883-9666-4f6d-9b4b-468972aa036a\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " Nov 03 06:00:46.018775 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.018683 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-dns-cert\") pod \"df6be883-9666-4f6d-9b4b-468972aa036a\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " Nov 03 06:00:46.018775 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.018704 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-kube-api-access-7z9k6\" (UniqueName: \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-kube-api-access-7z9k6\") pod \"df6be883-9666-4f6d-9b4b-468972aa036a\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " Nov 03 06:00:46.018775 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.018743 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-kubeconfig\") pod \"df6be883-9666-4f6d-9b4b-468972aa036a\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " Nov 03 06:00:46.018997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.018781 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-cacerts\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-cacerts\") pod \"df6be883-9666-4f6d-9b4b-468972aa036a\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " Nov 03 06:00:46.018997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.018863 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-istio-token\" (UniqueName: \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-istio-token\") pod \"df6be883-9666-4f6d-9b4b-468972aa036a\" (UID: \"df6be883-9666-4f6d-9b4b-468972aa036a\") " Nov 03 06:00:46.019105 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.019069 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-ca-configmap" pod "df6be883-9666-4f6d-9b4b-468972aa036a" (UID: "df6be883-9666-4f6d-9b4b-468972aa036a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:00:46.019214 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.019186 4855 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-ca-configmap\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:00:46.021248 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.021225 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6be883-9666-4f6d-9b4b-468972aa036a-local-certs" pod "df6be883-9666-4f6d-9b4b-468972aa036a" (UID: "df6be883-9666-4f6d-9b4b-468972aa036a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:00:46.021248 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.021228 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-kube-api-access-7z9k6" pod "df6be883-9666-4f6d-9b4b-468972aa036a" (UID: "df6be883-9666-4f6d-9b4b-468972aa036a"). InnerVolumeSpecName "kube-api-access-7z9k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:00:46.021451 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.021408 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-cacerts" pod "df6be883-9666-4f6d-9b4b-468972aa036a" (UID: "df6be883-9666-4f6d-9b4b-468972aa036a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:00:46.021451 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.021429 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-dns-cert" pod "df6be883-9666-4f6d-9b4b-468972aa036a" (UID: "df6be883-9666-4f6d-9b4b-468972aa036a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:00:46.021572 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.021499 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-kubeconfig" pod "df6be883-9666-4f6d-9b4b-468972aa036a" (UID: "df6be883-9666-4f6d-9b4b-468972aa036a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:00:46.021572 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.021513 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-istio-token" pod "df6be883-9666-4f6d-9b4b-468972aa036a" (UID: "df6be883-9666-4f6d-9b4b-468972aa036a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:00:46.120385 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.120347 4855 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-csr-dns-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:00:46.120385 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.120379 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z9k6\" (UniqueName: \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-kube-api-access-7z9k6\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:00:46.120385 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.120414 4855 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-istio-kubeconfig\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:00:46.120635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.120428 4855 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df6be883-9666-4f6d-9b4b-468972aa036a-cacerts\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:00:46.120635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.120436 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6be883-9666-4f6d-9b4b-468972aa036a-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:00:46.120635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.120444 4855 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6be883-9666-4f6d-9b4b-468972aa036a-local-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:00:46.656944 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.656850 4855 generic.go:356] "Generic (PLEG): container finished" podID="df6be883-9666-4f6d-9b4b-468972aa036a" containerID="d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90" exitCode=0 Nov 03 06:00:46.656944 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.656898 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" Nov 03 06:00:46.657445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.656938 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" event={"ID":"df6be883-9666-4f6d-9b4b-468972aa036a","Type":"ContainerDied","Data":"d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90"} Nov 03 06:00:46.657445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.656984 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k" event={"ID":"df6be883-9666-4f6d-9b4b-468972aa036a","Type":"ContainerDied","Data":"f1035839ec8c40834942c18695284d29f5d6022244f645092f1e865dcb577b71"} Nov 03 06:00:46.657445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.657003 4855 scope.go:122] "RemoveContainer" containerID="d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90" Nov 03 06:00:46.665173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.665155 4855 scope.go:122] "RemoveContainer" containerID="d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90" Nov 03 06:00:46.665476 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:00:46.665458 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90\": container with ID starting with d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90 not found: ID does not exist" containerID="d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90" Nov 03 06:00:46.665537 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.665485 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90"} err="failed to get container status \"d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90\": rpc error: code = NotFound desc = could not find container \"d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90\": container with ID starting with d9b780fbd37c056008a086ad0f11c6d9df934c89ea4174512b9ead30723e6b90 not found: ID does not exist" Nov 03 06:00:46.726515 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.726482 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k"] Nov 03 06:00:46.740429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:46.740398 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vjn6k"] Nov 03 06:00:48.331867 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:48.331832 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df6be883-9666-4f6d-9b4b-468972aa036a" path="/var/lib/kubelet/pods/df6be883-9666-4f6d-9b4b-468972aa036a/volumes" Nov 03 06:00:58.824525 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.824491 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-678d788c65-2vlhl"] Nov 03 06:00:58.824997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.824794 4855 cpu_manager.go:414] "containerMap: removing container" podUID="df6be883-9666-4f6d-9b4b-468972aa036a" containerName="discovery" Nov 03 06:00:58.824997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.824804 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df6be883-9666-4f6d-9b4b-468972aa036a" containerName="discovery" Nov 03 06:00:58.824997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.824886 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="df6be883-9666-4f6d-9b4b-468972aa036a" containerName="discovery" Nov 03 06:00:58.828224 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.828201 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:58.858955 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.858928 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4ffnq\"" Nov 03 06:00:58.858955 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.858943 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 06:00:58.859182 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.859015 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 06:00:58.859182 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.859034 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 06:00:58.859182 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.859101 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 06:00:58.859327 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.859285 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 06:00:58.860553 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.860524 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-678d788c65-2vlhl"] Nov 03 06:00:58.883507 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:00:58.883476 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:00:58.905033 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.905010 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-metrics-certs\") pod \"maas-controller-678d788c65-2vlhl\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:58.905148 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.905067 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-webhook-cert\") pod \"maas-controller-678d788c65-2vlhl\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:58.905211 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:58.905191 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdq8d\" (UniqueName: \"kubernetes.io/projected/878cafe2-86a7-4331-9a14-9874117a22db-kube-api-access-xdq8d\") pod \"maas-controller-678d788c65-2vlhl\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:59.006007 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:59.005974 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-webhook-cert\") pod \"maas-controller-678d788c65-2vlhl\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:59.006168 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:59.006032 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdq8d\" (UniqueName: \"kubernetes.io/projected/878cafe2-86a7-4331-9a14-9874117a22db-kube-api-access-xdq8d\") pod \"maas-controller-678d788c65-2vlhl\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:59.006168 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:59.006064 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-metrics-certs\") pod \"maas-controller-678d788c65-2vlhl\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:59.008427 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:59.008402 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-metrics-certs\") pod \"maas-controller-678d788c65-2vlhl\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:59.008547 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:59.008463 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-webhook-cert\") pod \"maas-controller-678d788c65-2vlhl\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:59.031969 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:59.031942 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdq8d\" (UniqueName: \"kubernetes.io/projected/878cafe2-86a7-4331-9a14-9874117a22db-kube-api-access-xdq8d\") pod \"maas-controller-678d788c65-2vlhl\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:59.140991 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:59.140911 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:00:59.438918 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:59.438893 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-678d788c65-2vlhl"] Nov 03 06:00:59.441117 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:00:59.441090 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878cafe2_86a7_4331_9a14_9874117a22db.slice/crio-2bcfca4b489c79dd151e8c80b66638298b9bda36d183721bfb2283bb399ac91e WatchSource:0}: Error finding container 2bcfca4b489c79dd151e8c80b66638298b9bda36d183721bfb2283bb399ac91e: Status 404 returned error can't find the container with id 2bcfca4b489c79dd151e8c80b66638298b9bda36d183721bfb2283bb399ac91e Nov 03 06:00:59.709023 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:00:59.708944 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-678d788c65-2vlhl" event={"ID":"878cafe2-86a7-4331-9a14-9874117a22db","Type":"ContainerStarted","Data":"2bcfca4b489c79dd151e8c80b66638298b9bda36d183721bfb2283bb399ac91e"} Nov 03 06:01:02.723546 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:02.723510 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-678d788c65-2vlhl" event={"ID":"878cafe2-86a7-4331-9a14-9874117a22db","Type":"ContainerStarted","Data":"743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d"} Nov 03 06:01:02.723926 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:02.723612 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:01:02.770343 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:02.770294 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-678d788c65-2vlhl" podStartSLOduration=2.550151081 podStartE2EDuration="4.770280429s" totalImagesPullingTime="2.220129348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:01:02.768691804 +0000 UTC m=+350.577079072" watchObservedRunningTime="2026-11-03 06:01:02.770280429 +0000 UTC m=+350.578667697" Nov 03 06:01:03.548525 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.548488 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s"] Nov 03 06:01:03.551940 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.551921 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.570177 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.570153 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 06:01:03.583749 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.583724 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-q6s8n\"" Nov 03 06:01:03.585645 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.585615 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s"] Nov 03 06:01:03.590947 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.590915 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 06:01:03.591104 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.590979 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 06:01:03.644763 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.644736 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e5babcd3-d9e7-409d-a677-fa56d8d024c2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.644912 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.644780 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9vs\" (UniqueName: \"kubernetes.io/projected/e5babcd3-d9e7-409d-a677-fa56d8d024c2-kube-api-access-nj9vs\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.644912 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.644807 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5babcd3-d9e7-409d-a677-fa56d8d024c2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.644982 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.644964 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e5babcd3-d9e7-409d-a677-fa56d8d024c2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.746125 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.746093 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e5babcd3-d9e7-409d-a677-fa56d8d024c2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.746654 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.746141 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e5babcd3-d9e7-409d-a677-fa56d8d024c2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.746654 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.746249 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9vs\" (UniqueName: \"kubernetes.io/projected/e5babcd3-d9e7-409d-a677-fa56d8d024c2-kube-api-access-nj9vs\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.746654 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.746314 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5babcd3-d9e7-409d-a677-fa56d8d024c2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.747195 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.747170 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5babcd3-d9e7-409d-a677-fa56d8d024c2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.747305 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.747291 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e5babcd3-d9e7-409d-a677-fa56d8d024c2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.748563 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.748544 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e5babcd3-d9e7-409d-a677-fa56d8d024c2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.771078 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.771049 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9vs\" (UniqueName: \"kubernetes.io/projected/e5babcd3-d9e7-409d-a677-fa56d8d024c2-kube-api-access-nj9vs\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gth2s\" (UID: \"e5babcd3-d9e7-409d-a677-fa56d8d024c2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:03.865497 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:03.865417 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:04.162606 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:04.162520 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s"] Nov 03 06:01:04.165756 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:01:04.165713 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5babcd3_d9e7_409d_a677_fa56d8d024c2.slice/crio-2323859cad0e24f12f4ccc45a42453c523f63bdeb0c4f47ce89fc9d2650ea346 WatchSource:0}: Error finding container 2323859cad0e24f12f4ccc45a42453c523f63bdeb0c4f47ce89fc9d2650ea346: Status 404 returned error can't find the container with id 2323859cad0e24f12f4ccc45a42453c523f63bdeb0c4f47ce89fc9d2650ea346 Nov 03 06:01:04.733712 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:04.733678 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" event={"ID":"e5babcd3-d9e7-409d-a677-fa56d8d024c2","Type":"ContainerStarted","Data":"2323859cad0e24f12f4ccc45a42453c523f63bdeb0c4f47ce89fc9d2650ea346"} Nov 03 06:01:13.730806 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:13.730772 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:01:17.805242 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:17.805205 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" event={"ID":"e5babcd3-d9e7-409d-a677-fa56d8d024c2","Type":"ContainerStarted","Data":"fd5fafbfcb8512a5bfefddeff057e60e7ba593b276e885bd49737e17d249a21c"} Nov 03 06:01:17.805660 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:17.805348 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:17.850498 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:17.850447 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" podStartSLOduration=2.282811437 podStartE2EDuration="14.850433469s" totalImagesPullingTime="12.567622032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:01:17.849512454 +0000 UTC m=+365.657899722" watchObservedRunningTime="2026-11-03 06:01:17.850433469 +0000 UTC m=+365.658820736" Nov 03 06:01:28.814622 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:28.814591 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s" Nov 03 06:01:28.886405 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:01:28.886353 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:01:40.997176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:40.997142 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:01:49.481324 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.481294 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk"] Nov 03 06:01:49.529642 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.529613 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk"] Nov 03 06:01:49.529789 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.529729 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.549156 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.549134 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 06:01:49.562694 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.562673 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 06:01:49.627121 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.627094 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 06:01:49.627692 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.627677 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k6kkr\"" Nov 03 06:01:49.627775 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.627762 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 06:01:49.641172 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.641148 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-maas-api-tls\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.641271 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.641185 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-openshift-service-ca\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.641331 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.641311 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-metrics-certs\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.641416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.641380 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhd2\" (UniqueName: \"kubernetes.io/projected/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-kube-api-access-lfhd2\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.741810 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.741737 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-maas-api-tls\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.741810 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.741773 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-openshift-service-ca\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.741810 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.741806 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-metrics-certs\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.742047 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.741841 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhd2\" (UniqueName: \"kubernetes.io/projected/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-kube-api-access-lfhd2\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.742421 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.742377 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-openshift-service-ca\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.744324 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.744306 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-metrics-certs\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.744423 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.744372 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-maas-api-tls\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.766016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.765989 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhd2\" (UniqueName: \"kubernetes.io/projected/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-kube-api-access-lfhd2\") pod \"maas-api-6fd5d8948d-sgjnk\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:49.843635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:49.843603 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:50.140988 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:50.140958 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk"] Nov 03 06:01:50.144124 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:01:50.144077 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061fb5a9_3b3a_44e8_90b7_ab7d51b28496.slice/crio-31f02660ef8e2e7adcb0c0671d36d5a9988df9ea22d74bdea97f350e0ffa8345 WatchSource:0}: Error finding container 31f02660ef8e2e7adcb0c0671d36d5a9988df9ea22d74bdea97f350e0ffa8345: Status 404 returned error can't find the container with id 31f02660ef8e2e7adcb0c0671d36d5a9988df9ea22d74bdea97f350e0ffa8345 Nov 03 06:01:50.935019 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:50.934974 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" event={"ID":"061fb5a9-3b3a-44e8-90b7-ab7d51b28496","Type":"ContainerStarted","Data":"31f02660ef8e2e7adcb0c0671d36d5a9988df9ea22d74bdea97f350e0ffa8345"} Nov 03 06:01:51.941237 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:51.941201 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" event={"ID":"061fb5a9-3b3a-44e8-90b7-ab7d51b28496","Type":"ContainerStarted","Data":"7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c"} Nov 03 06:01:51.941653 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:51.941271 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:51.985732 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:51.985684 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" podStartSLOduration=1.7238749169999998 podStartE2EDuration="2.985670491s" totalImagesPullingTime="1.261795574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:01:51.985417897 +0000 UTC m=+399.793805166" watchObservedRunningTime="2026-11-03 06:01:51.985670491 +0000 UTC m=+399.794057759" Nov 03 06:01:57.951816 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:01:57.951745 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:01:58.885716 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:01:58.885674 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:02:28.882984 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:02:28.882949 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:02:41.842479 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:41.842439 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:02:42.443106 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.443069 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9"] Nov 03 06:02:42.448485 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.448457 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.478824 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.478788 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 06:02:42.478824 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.478814 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t4d9g\"" Nov 03 06:02:42.480193 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.480174 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9"] Nov 03 06:02:42.480853 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.480839 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 06:02:42.480904 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.480870 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 06:02:42.561518 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.561485 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.561518 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.561523 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.561781 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.561555 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.561781 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.561621 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhctt\" (UniqueName: \"kubernetes.io/projected/ee25c503-9d87-47a1-b630-e0d6d24e1c18-kube-api-access-rhctt\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.561781 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.561676 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.561781 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.561734 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee25c503-9d87-47a1-b630-e0d6d24e1c18-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.662380 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.662346 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee25c503-9d87-47a1-b630-e0d6d24e1c18-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.662559 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.662415 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.662559 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.662448 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.662629 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.662583 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.662629 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.662621 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhctt\" (UniqueName: \"kubernetes.io/projected/ee25c503-9d87-47a1-b630-e0d6d24e1c18-kube-api-access-rhctt\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.662715 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.662665 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.662864 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.662844 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.662937 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.662895 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.662997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.662959 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.664810 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.664782 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee25c503-9d87-47a1-b630-e0d6d24e1c18-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.664937 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.664919 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee25c503-9d87-47a1-b630-e0d6d24e1c18-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.690357 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.690336 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhctt\" (UniqueName: \"kubernetes.io/projected/ee25c503-9d87-47a1-b630-e0d6d24e1c18-kube-api-access-rhctt\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dhcm9\" (UID: \"ee25c503-9d87-47a1-b630-e0d6d24e1c18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:42.762080 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:42.762003 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:43.060850 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:43.060763 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9"] Nov 03 06:02:43.063748 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:02:43.063722 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee25c503_9d87_47a1_b630_e0d6d24e1c18.slice/crio-857d197984a00ec3dde4c1ddde501823570d781996ab599af629990c7d28854d WatchSource:0}: Error finding container 857d197984a00ec3dde4c1ddde501823570d781996ab599af629990c7d28854d: Status 404 returned error can't find the container with id 857d197984a00ec3dde4c1ddde501823570d781996ab599af629990c7d28854d Nov 03 06:02:43.158593 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:43.158563 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" event={"ID":"ee25c503-9d87-47a1-b630-e0d6d24e1c18","Type":"ContainerStarted","Data":"857d197984a00ec3dde4c1ddde501823570d781996ab599af629990c7d28854d"} Nov 03 06:02:44.872065 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:44.872020 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:02:45.168032 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:45.167950 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" event={"ID":"ee25c503-9d87-47a1-b630-e0d6d24e1c18","Type":"ContainerStarted","Data":"9da067cd974f8469d128aaecf3144f4ce00800ae426170b719e5f3d974b9551f"} Nov 03 06:02:45.215187 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:45.215129 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" podStartSLOduration=2.157147432 podStartE2EDuration="3.215115287s" totalImagesPullingTime="1.057967855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:02:45.215071184 +0000 UTC m=+453.023458452" watchObservedRunningTime="2026-11-03 06:02:45.215115287 +0000 UTC m=+453.023502554" Nov 03 06:02:45.762815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:45.762782 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:46.718220 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.718190 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2"] Nov 03 06:02:46.721680 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.721664 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.751696 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.751674 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 06:02:46.753270 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.753251 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2"] Nov 03 06:02:46.800529 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.800495 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.800701 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.800554 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.800701 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.800600 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.800701 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.800666 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-568wm\" (UniqueName: \"kubernetes.io/projected/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-kube-api-access-568wm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.800873 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.800736 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.800873 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.800772 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.901863 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.901823 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.902055 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.901882 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-568wm\" (UniqueName: \"kubernetes.io/projected/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-kube-api-access-568wm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.902055 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.901917 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.902055 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.901936 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.902055 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.901965 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.902055 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.901999 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.902611 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.902585 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.902770 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.902750 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.902840 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.902792 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.904306 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.904280 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.904761 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.904738 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:46.927427 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:46.927344 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-568wm\" (UniqueName: \"kubernetes.io/projected/5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d-kube-api-access-568wm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2\" (UID: \"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:47.035352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:47.035267 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:47.336445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:47.336417 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2"] Nov 03 06:02:47.338604 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:02:47.338578 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed1974f_ec09_4cd4_b5e9_5e7211f13b9d.slice/crio-69fc84f4a5fd39dbded5610f2e1883b685f3b12e2f380805a45944bb900aa8ac WatchSource:0}: Error finding container 69fc84f4a5fd39dbded5610f2e1883b685f3b12e2f380805a45944bb900aa8ac: Status 404 returned error can't find the container with id 69fc84f4a5fd39dbded5610f2e1883b685f3b12e2f380805a45944bb900aa8ac Nov 03 06:02:48.181642 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.181607 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" event={"ID":"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d","Type":"ContainerStarted","Data":"474d0c8f3267949f66de4ef069d223125a3b4c8e105bc3dbda71d705b2c6067c"} Nov 03 06:02:48.182034 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.181661 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" event={"ID":"5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d","Type":"ContainerStarted","Data":"69fc84f4a5fd39dbded5610f2e1883b685f3b12e2f380805a45944bb900aa8ac"} Nov 03 06:02:48.228005 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.227958 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" podStartSLOduration=2.047279433 podStartE2EDuration="2.227944064s" totalImagesPullingTime="180.664631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:02:48.226982976 +0000 UTC m=+456.035370243" watchObservedRunningTime="2026-11-03 06:02:48.227944064 +0000 UTC m=+456.036331331" Nov 03 06:02:48.620238 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.620209 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ds4ns"] Nov 03 06:02:48.623952 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.623933 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.645162 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.645137 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ds4ns"] Nov 03 06:02:48.656695 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.656674 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 06:02:48.718687 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.718648 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.718687 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.718685 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.718898 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.718789 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqjhm\" (UniqueName: \"kubernetes.io/projected/a65e242d-508a-4b6e-860f-f9095ccb686b-kube-api-access-dqjhm\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.718898 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.718819 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a65e242d-508a-4b6e-860f-f9095ccb686b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.718898 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.718856 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.719021 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.718976 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-home\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.819780 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.819745 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.819780 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.819780 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.819967 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.819837 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqjhm\" (UniqueName: \"kubernetes.io/projected/a65e242d-508a-4b6e-860f-f9095ccb686b-kube-api-access-dqjhm\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.819967 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.819875 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a65e242d-508a-4b6e-860f-f9095ccb686b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.819967 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.819922 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.819967 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.819963 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-home\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.820228 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.820205 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.820299 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.820252 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-home\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.820401 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.820366 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.822062 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.822038 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a65e242d-508a-4b6e-860f-f9095ccb686b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.822311 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.822292 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a65e242d-508a-4b6e-860f-f9095ccb686b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.846409 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.846360 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqjhm\" (UniqueName: \"kubernetes.io/projected/a65e242d-508a-4b6e-860f-f9095ccb686b-kube-api-access-dqjhm\") pod \"echo-simulator-kserve-64b6f8f96c-ds4ns\" (UID: \"a65e242d-508a-4b6e-860f-f9095ccb686b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:48.938209 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:48.938137 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:49.035839 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:49.035807 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:49.237206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:49.237183 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ds4ns"] Nov 03 06:02:49.239706 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:02:49.239675 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65e242d_508a_4b6e_860f_f9095ccb686b.slice/crio-f7def0ee1c992e15e1021decc225e3716f1527f655d2edb34e394c98d8189b18 WatchSource:0}: Error finding container f7def0ee1c992e15e1021decc225e3716f1527f655d2edb34e394c98d8189b18: Status 404 returned error can't find the container with id f7def0ee1c992e15e1021decc225e3716f1527f655d2edb34e394c98d8189b18 Nov 03 06:02:49.557916 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:49.557880 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:02:50.192070 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:50.192033 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" event={"ID":"a65e242d-508a-4b6e-860f-f9095ccb686b","Type":"ContainerStarted","Data":"f7def0ee1c992e15e1021decc225e3716f1527f655d2edb34e394c98d8189b18"} Nov 03 06:02:51.203663 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:51.203619 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" event={"ID":"a65e242d-508a-4b6e-860f-f9095ccb686b","Type":"ContainerStarted","Data":"b2e134a16d41c9ae5f440ef26e90e33a5cffe41f8526c17fce54dea8035504e1"} Nov 03 06:02:51.249907 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:51.249854 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" podStartSLOduration=1.725391499 podStartE2EDuration="3.24983995s" totalImagesPullingTime="1.524448451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:02:51.248836497 +0000 UTC m=+459.057223764" watchObservedRunningTime="2026-11-03 06:02:51.24983995 +0000 UTC m=+459.058227219" Nov 03 06:02:51.938545 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:51.938505 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:52.098623 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.098590 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5"] Nov 03 06:02:52.102098 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.102076 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.122541 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.122512 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5"] Nov 03 06:02:52.135008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.134977 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 06:02:52.250245 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.250157 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.250245 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.250213 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec588a2d-3f62-4256-afed-c86041c5a59b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.250670 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.250315 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.250670 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.250341 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xvh\" (UniqueName: \"kubernetes.io/projected/ec588a2d-3f62-4256-afed-c86041c5a59b-kube-api-access-h9xvh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.250670 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.250382 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.250670 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.250445 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.351457 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.351424 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.351646 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.351484 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec588a2d-3f62-4256-afed-c86041c5a59b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.351646 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.351543 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.351795 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.351662 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xvh\" (UniqueName: \"kubernetes.io/projected/ec588a2d-3f62-4256-afed-c86041c5a59b-kube-api-access-h9xvh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.351795 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.351767 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.351895 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.351798 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.351895 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.351821 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.352156 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.352131 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.352233 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.352170 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.353835 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.353806 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec588a2d-3f62-4256-afed-c86041c5a59b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.354113 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.354095 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec588a2d-3f62-4256-afed-c86041c5a59b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.376490 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.376465 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xvh\" (UniqueName: \"kubernetes.io/projected/ec588a2d-3f62-4256-afed-c86041c5a59b-kube-api-access-h9xvh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5\" (UID: \"ec588a2d-3f62-4256-afed-c86041c5a59b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.426315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.426282 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:52.725839 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.725806 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5"] Nov 03 06:02:52.728756 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:02:52.728731 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec588a2d_3f62_4256_afed_c86041c5a59b.slice/crio-eb800351ba57c499c9c3fa8466b0fa722edfa1bd0ebb999a4d520f843f27bac9 WatchSource:0}: Error finding container eb800351ba57c499c9c3fa8466b0fa722edfa1bd0ebb999a4d520f843f27bac9: Status 404 returned error can't find the container with id eb800351ba57c499c9c3fa8466b0fa722edfa1bd0ebb999a4d520f843f27bac9 Nov 03 06:02:52.762418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.762373 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:52.774802 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:52.774779 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:53.213664 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:53.213618 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" event={"ID":"ec588a2d-3f62-4256-afed-c86041c5a59b","Type":"ContainerStarted","Data":"8890e1fbfa0096fe785e3dfbd564548c1bc5110ab9c6751cc1f09d617e4c3ac3"} Nov 03 06:02:53.213823 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:53.213676 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" event={"ID":"ec588a2d-3f62-4256-afed-c86041c5a59b","Type":"ContainerStarted","Data":"eb800351ba57c499c9c3fa8466b0fa722edfa1bd0ebb999a4d520f843f27bac9"} Nov 03 06:02:53.224939 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:53.224911 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dhcm9" Nov 03 06:02:53.259161 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:53.259108 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podStartSLOduration=1.089624053 podStartE2EDuration="1.259093651s" totalImagesPullingTime="169.469598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:02:53.257883724 +0000 UTC m=+461.066270993" watchObservedRunningTime="2026-11-03 06:02:53.259093651 +0000 UTC m=+461.067480920" Nov 03 06:02:53.426768 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:53.426734 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:02:57.035982 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:57.035941 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:57.048466 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:57.048437 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:57.243945 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:57.243911 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2" Nov 03 06:02:57.841187 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:57.841154 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:02:58.883051 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:02:58.883016 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:02:58.938562 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:58.938531 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:58.951201 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:58.951176 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:02:59.253018 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:02:59.252943 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ds4ns" Nov 03 06:03:02.427561 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:02.427521 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:03:02.440229 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:02.440205 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:03:03.271480 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:03.271452 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:03:10.871069 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.871032 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj"] Nov 03 06:03:10.874816 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.874795 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:10.894277 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.894257 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j7v45\"" Nov 03 06:03:10.906595 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.906574 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj"] Nov 03 06:03:10.910234 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.910212 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:10.910354 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.910239 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:10.910354 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.910266 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjn7f\" (UniqueName: \"kubernetes.io/projected/91014878-7b02-4ef8-8921-5bcef4cefa7b-kube-api-access-gjn7f\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:10.910354 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.910287 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:10.910553 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.910356 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:10.910553 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.910498 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:10.910553 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.910534 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:10.910763 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.910556 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:10.910763 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:10.910580 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91014878-7b02-4ef8-8921-5bcef4cefa7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.011971 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.011939 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.011979 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012006 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91014878-7b02-4ef8-8921-5bcef4cefa7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012041 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012058 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012083 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjn7f\" (UniqueName: \"kubernetes.io/projected/91014878-7b02-4ef8-8921-5bcef4cefa7b-kube-api-access-gjn7f\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012109 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012131 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012172 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012688 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012491 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012688 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012524 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012799 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012711 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.012856 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.012818 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.013209 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.013184 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91014878-7b02-4ef8-8921-5bcef4cefa7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.014459 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.014436 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.014801 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.014783 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.037304 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.037272 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91014878-7b02-4ef8-8921-5bcef4cefa7b-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.037452 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.037329 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjn7f\" (UniqueName: \"kubernetes.io/projected/91014878-7b02-4ef8-8921-5bcef4cefa7b-kube-api-access-gjn7f\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nnlxj\" (UID: \"91014878-7b02-4ef8-8921-5bcef4cefa7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.188502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.188413 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:11.521188 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:11.521101 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj"] Nov 03 06:03:11.524053 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:03:11.524028 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91014878_7b02_4ef8_8921_5bcef4cefa7b.slice/crio-bde5c4a03db4517e33b950f7f3959f67cdf24842021411fbc46536680a176dbe WatchSource:0}: Error finding container bde5c4a03db4517e33b950f7f3959f67cdf24842021411fbc46536680a176dbe: Status 404 returned error can't find the container with id bde5c4a03db4517e33b950f7f3959f67cdf24842021411fbc46536680a176dbe Nov 03 06:03:12.301229 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:12.301189 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" event={"ID":"91014878-7b02-4ef8-8921-5bcef4cefa7b","Type":"ContainerStarted","Data":"bde5c4a03db4517e33b950f7f3959f67cdf24842021411fbc46536680a176dbe"} Nov 03 06:03:14.074383 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:14.074352 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:03:14.074660 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:14.074439 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:03:14.074660 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:14.074470 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:03:14.312094 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:14.312056 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" event={"ID":"91014878-7b02-4ef8-8921-5bcef4cefa7b","Type":"ContainerStarted","Data":"a43408c78d0c4267aa00494773cbbd311b6a0678dd31e9d8929aad1a4d1aaf73"} Nov 03 06:03:14.375024 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:14.374924 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" podStartSLOduration=1.827420347 podStartE2EDuration="4.374911769s" totalImagesPullingTime="2.547491422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:03:14.373766614 +0000 UTC m=+482.182153882" watchObservedRunningTime="2026-11-03 06:03:14.374911769 +0000 UTC m=+482.183299036" Nov 03 06:03:15.188866 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:15.188825 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:15.190265 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:15.190242 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nnlxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 06:03:15.190342 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:15.190291 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" podUID="91014878-7b02-4ef8-8921-5bcef4cefa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 06:03:16.189716 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:16.189681 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nnlxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 06:03:16.190099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:16.189731 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" podUID="91014878-7b02-4ef8-8921-5bcef4cefa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 06:03:17.189246 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:17.189209 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nnlxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 06:03:17.189470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:17.189263 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" podUID="91014878-7b02-4ef8-8921-5bcef4cefa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 06:03:18.189320 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:18.189287 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nnlxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 06:03:18.189913 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:18.189347 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" podUID="91014878-7b02-4ef8-8921-5bcef4cefa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 06:03:19.189177 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:19.189140 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nnlxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 06:03:19.189347 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:19.189225 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" podUID="91014878-7b02-4ef8-8921-5bcef4cefa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 06:03:20.189081 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:20.189040 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nnlxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 06:03:20.189259 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:20.189124 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" podUID="91014878-7b02-4ef8-8921-5bcef4cefa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 06:03:21.189190 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:21.189143 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nnlxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 06:03:21.189739 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:21.189200 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" podUID="91014878-7b02-4ef8-8921-5bcef4cefa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 06:03:23.230558 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:23.230502 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nnlxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:03:23.231079 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:23.230590 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" podUID="91014878-7b02-4ef8-8921-5bcef4cefa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:03:24.270654 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:24.270570 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nnlxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:03:24.271019 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:24.270643 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" podUID="91014878-7b02-4ef8-8921-5bcef4cefa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:03:24.349826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:24.349796 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:24.350181 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:24.350153 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:24.351080 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:24.351050 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nnlxj" Nov 03 06:03:25.917340 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.917303 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p"] Nov 03 06:03:25.921344 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.921321 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:25.951695 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.951667 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p"] Nov 03 06:03:25.965325 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.965294 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:25.965517 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.965364 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:25.965517 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.965405 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17697121-d871-43b4-b1e8-5e74fb513cd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:25.965517 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.965425 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rd79\" (UniqueName: \"kubernetes.io/projected/17697121-d871-43b4-b1e8-5e74fb513cd5-kube-api-access-4rd79\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:25.965517 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.965450 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:25.965517 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.965482 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:25.965517 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.965508 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:25.965862 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.965525 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:25.965862 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:25.965542 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.065960 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.065918 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066166 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.065974 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066166 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066010 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066166 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066036 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066166 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066058 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066372 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066218 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066372 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066338 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066513 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066375 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17697121-d871-43b4-b1e8-5e74fb513cd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066513 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066431 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rd79\" (UniqueName: \"kubernetes.io/projected/17697121-d871-43b4-b1e8-5e74fb513cd5-kube-api-access-4rd79\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066513 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066459 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066518 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066571 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.066658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.066612 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.067059 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.067040 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17697121-d871-43b4-b1e8-5e74fb513cd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.068552 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.068524 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.068717 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.068702 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.090910 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.090880 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17697121-d871-43b4-b1e8-5e74fb513cd5-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.091001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.090945 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rd79\" (UniqueName: \"kubernetes.io/projected/17697121-d871-43b4-b1e8-5e74fb513cd5-kube-api-access-4rd79\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ssx5p\" (UID: \"17697121-d871-43b4-b1e8-5e74fb513cd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.234237 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.234153 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:26.552663 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.552633 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p"] Nov 03 06:03:26.555317 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:03:26.555288 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17697121_d871_43b4_b1e8_5e74fb513cd5.slice/crio-6f6d39a2f8e48e8d7261538d895e4e65754c6069b6a10027f3064811a0c55fe3 WatchSource:0}: Error finding container 6f6d39a2f8e48e8d7261538d895e4e65754c6069b6a10027f3064811a0c55fe3: Status 404 returned error can't find the container with id 6f6d39a2f8e48e8d7261538d895e4e65754c6069b6a10027f3064811a0c55fe3 Nov 03 06:03:26.558307 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.558272 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:03:26.558407 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.558341 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:03:26.558407 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:26.558370 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:03:27.394099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:27.394065 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" event={"ID":"17697121-d871-43b4-b1e8-5e74fb513cd5","Type":"ContainerStarted","Data":"704864cb722f791a2a888d330e16b2e1f95d90fe13daa835a92814dd2072b0f4"} Nov 03 06:03:27.394516 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:27.394114 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" event={"ID":"17697121-d871-43b4-b1e8-5e74fb513cd5","Type":"ContainerStarted","Data":"6f6d39a2f8e48e8d7261538d895e4e65754c6069b6a10027f3064811a0c55fe3"} Nov 03 06:03:27.457215 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:27.457147 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" podStartSLOduration=2.457128918 podStartE2EDuration="2.457128918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:03:27.456076007 +0000 UTC m=+495.264463275" watchObservedRunningTime="2026-11-03 06:03:27.457128918 +0000 UTC m=+495.265516186" Nov 03 06:03:28.234737 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:28.234700 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:28.244518 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:28.244485 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ssx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 06:03:28.244754 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:28.244729 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" podUID="17697121-d871-43b4-b1e8-5e74fb513cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 06:03:28.882325 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:03:28.882293 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:03:29.246482 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:29.241779 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ssx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 06:03:29.246482 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:29.241845 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" podUID="17697121-d871-43b4-b1e8-5e74fb513cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 06:03:30.240733 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:30.240697 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ssx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 06:03:30.241313 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:30.241281 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" podUID="17697121-d871-43b4-b1e8-5e74fb513cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 06:03:31.241415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:31.240416 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ssx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 06:03:31.241415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:31.240479 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" podUID="17697121-d871-43b4-b1e8-5e74fb513cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 06:03:32.253544 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:32.253506 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ssx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 06:03:32.254137 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:32.254106 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" podUID="17697121-d871-43b4-b1e8-5e74fb513cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 06:03:33.234675 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:33.234644 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ssx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 06:03:33.234863 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:33.234703 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" podUID="17697121-d871-43b4-b1e8-5e74fb513cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 06:03:34.746572 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:34.746541 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:34.747050 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:34.746873 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:34.747805 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:03:34.747781 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" Nov 03 06:03:58.882371 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:03:58.882332 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:04:08.253475 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.253439 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml"] Nov 03 06:04:08.257442 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.257418 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.274608 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.274583 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9688d8-openshift-default-dockercfg-ldnhj\"" Nov 03 06:04:08.289893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.289863 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml"] Nov 03 06:04:08.343856 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.343825 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67kbq\" (UniqueName: \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-kube-api-access-67kbq\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.344030 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.343882 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.344093 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.344017 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.344093 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.344051 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.344206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.344165 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.344281 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.344261 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.344452 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.344409 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5507309e-a297-480d-bd20-4d3914fe9cae-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.344547 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.344530 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.344589 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.344574 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5507309e-a297-480d-bd20-4d3914fe9cae-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.417436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.417374 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t"] Nov 03 06:04:08.421004 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.420979 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.442245 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.442216 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t"] Nov 03 06:04:08.445852 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.445826 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.445977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.445869 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.445977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.445899 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.446095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446034 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.446095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446075 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5507309e-a297-480d-bd20-4d3914fe9cae-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.446204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446103 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.446204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446130 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d926084c-dafb-4fd7-adc3-928dc9036a92-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.446204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446156 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx78j\" (UniqueName: \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-kube-api-access-lx78j\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.446346 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446200 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67kbq\" (UniqueName: \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-kube-api-access-67kbq\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.446346 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446226 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.446346 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446260 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.446346 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446290 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.446346 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446311 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.446639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446349 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.446639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446448 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.446639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446475 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.446639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446510 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.446639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446546 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5507309e-a297-480d-bd20-4d3914fe9cae-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.446639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446582 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.446639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.446611 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.447281 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.447245 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.447380 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.447300 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.448323 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.448299 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5507309e-a297-480d-bd20-4d3914fe9cae-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.449636 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.449612 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5507309e-a297-480d-bd20-4d3914fe9cae-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.449636 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.449630 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.477885 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.477861 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.478002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.477929 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67kbq\" (UniqueName: \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-kube-api-access-67kbq\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.485221 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.485191 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-be1a66-openshift-default-dockercfg-kn8kd\"" Nov 03 06:04:08.548056 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548032 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548063 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548097 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548138 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548170 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548318 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d926084c-dafb-4fd7-adc3-928dc9036a92-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548361 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx78j\" (UniqueName: \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-kube-api-access-lx78j\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548437 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548616 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548468 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548616 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548586 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548717 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548614 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548717 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548635 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.548820 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.548773 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.549066 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.549042 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d926084c-dafb-4fd7-adc3-928dc9036a92-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.550651 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.550628 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.550800 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.550785 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.576814 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.576780 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:08.585283 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.585261 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.586445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.586370 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx78j\" (UniqueName: \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-kube-api-access-lx78j\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.713705 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.713673 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b"] Nov 03 06:04:08.724991 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.724958 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.735993 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.735540 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:08.743917 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.743890 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b"] Nov 03 06:04:08.751604 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.751575 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.751714 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.751642 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.751714 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.751702 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.751825 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.751786 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6bql\" (UniqueName: \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-kube-api-access-g6bql\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.751879 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.751829 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299590cc-de3d-4408-8653-320887002525-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.751937 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.751888 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.751990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.751953 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.752042 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.751999 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.752089 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.752050 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299590cc-de3d-4408-8653-320887002525-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.754963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.754939 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ba6892-openshift-default-dockercfg-mtf55\"" Nov 03 06:04:08.852827 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.852736 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.852827 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.852781 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.852827 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.852825 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.853099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.852871 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299590cc-de3d-4408-8653-320887002525-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.853099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.852924 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.853099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.853003 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.853099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.853031 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.853099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.853067 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6bql\" (UniqueName: \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-kube-api-access-g6bql\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.853343 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.853106 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299590cc-de3d-4408-8653-320887002525-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.853343 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.853251 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.853486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.853368 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.853667 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.853646 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.854102 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.854080 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299590cc-de3d-4408-8653-320887002525-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.854637 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.854611 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.855874 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.855849 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.856267 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.856248 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299590cc-de3d-4408-8653-320887002525-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.894811 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.894772 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.897114 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.897088 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6bql\" (UniqueName: \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-kube-api-access-g6bql\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:08.910589 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.910568 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml"] Nov 03 06:04:08.913404 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:04:08.913361 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5507309e_a297_480d_bd20_4d3914fe9cae.slice/crio-5084a3f4c4486861323e9dfa81ba549a751f7e75d5a691ca39955ff85da22a7f WatchSource:0}: Error finding container 5084a3f4c4486861323e9dfa81ba549a751f7e75d5a691ca39955ff85da22a7f: Status 404 returned error can't find the container with id 5084a3f4c4486861323e9dfa81ba549a751f7e75d5a691ca39955ff85da22a7f Nov 03 06:04:08.915711 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.915675 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:08.915825 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.915805 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:08.915895 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:08.915879 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:09.040058 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.040021 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:09.049137 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.044643 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t"] Nov 03 06:04:09.055560 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:04:09.055526 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd926084c_dafb_4fd7_adc3_928dc9036a92.slice/crio-1c7a4b345b3a71b47dfa6ac3a5b02c7178f210995fa0d279f1645aa0cb9cc7aa WatchSource:0}: Error finding container 1c7a4b345b3a71b47dfa6ac3a5b02c7178f210995fa0d279f1645aa0cb9cc7aa: Status 404 returned error can't find the container with id 1c7a4b345b3a71b47dfa6ac3a5b02c7178f210995fa0d279f1645aa0cb9cc7aa Nov 03 06:04:09.062762 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.061951 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:09.062762 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.062146 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:09.062762 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.062199 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:09.376532 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.376338 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b"] Nov 03 06:04:09.379844 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:04:09.379816 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299590cc_de3d_4408_8653_320887002525.slice/crio-cacb752163b30657e9da9ad2413c2a43b2d57539d0e763e687098f2fd229032d WatchSource:0}: Error finding container cacb752163b30657e9da9ad2413c2a43b2d57539d0e763e687098f2fd229032d: Status 404 returned error can't find the container with id cacb752163b30657e9da9ad2413c2a43b2d57539d0e763e687098f2fd229032d Nov 03 06:04:09.382224 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.382195 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:09.382296 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.382258 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:09.382296 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.382284 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:09.585172 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.585112 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" event={"ID":"299590cc-de3d-4408-8653-320887002525","Type":"ContainerStarted","Data":"6bafdf370835679a5d1fa10ac864ba0fef9f5d1933cec2f0e1c95b6c7d86e03b"} Nov 03 06:04:09.585172 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.585175 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" event={"ID":"299590cc-de3d-4408-8653-320887002525","Type":"ContainerStarted","Data":"cacb752163b30657e9da9ad2413c2a43b2d57539d0e763e687098f2fd229032d"} Nov 03 06:04:09.586879 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.586854 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" event={"ID":"5507309e-a297-480d-bd20-4d3914fe9cae","Type":"ContainerStarted","Data":"5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53"} Nov 03 06:04:09.587016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.586896 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" event={"ID":"5507309e-a297-480d-bd20-4d3914fe9cae","Type":"ContainerStarted","Data":"5084a3f4c4486861323e9dfa81ba549a751f7e75d5a691ca39955ff85da22a7f"} Nov 03 06:04:09.588331 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.588310 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" event={"ID":"d926084c-dafb-4fd7-adc3-928dc9036a92","Type":"ContainerStarted","Data":"4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218"} Nov 03 06:04:09.588429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.588344 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" event={"ID":"d926084c-dafb-4fd7-adc3-928dc9036a92","Type":"ContainerStarted","Data":"1c7a4b345b3a71b47dfa6ac3a5b02c7178f210995fa0d279f1645aa0cb9cc7aa"} Nov 03 06:04:09.636021 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.635929 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podStartSLOduration=1.635916191 podStartE2EDuration="1.635916191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:09.634673286 +0000 UTC m=+537.443060554" watchObservedRunningTime="2026-11-03 06:04:09.635916191 +0000 UTC m=+537.444303526" Nov 03 06:04:09.683591 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.683543 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" podStartSLOduration=1.6835290299999999 podStartE2EDuration="1.68352903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:09.682617842 +0000 UTC m=+537.491005110" watchObservedRunningTime="2026-11-03 06:04:09.68352903 +0000 UTC m=+537.491916298" Nov 03 06:04:09.731076 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.731028 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" podStartSLOduration=1.7310132299999998 podStartE2EDuration="1.73101323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:09.729490368 +0000 UTC m=+537.537877647" watchObservedRunningTime="2026-11-03 06:04:09.73101323 +0000 UTC m=+537.539400497" Nov 03 06:04:09.735975 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:09.735948 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:10.041179 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:10.041100 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:10.046646 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:10.046622 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:10.577291 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:10.577257 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:10.583189 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:10.583162 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:10.595221 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:10.595184 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:10.595443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:10.595251 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:10.596162 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:10.596138 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:10.596373 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:10.596359 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:10.742348 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:10.742321 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:11.605075 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:11.603667 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:11.605075 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:11.603903 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:13.373075 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.373038 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5"] Nov 03 06:04:13.377774 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.377758 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.401268 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.401171 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.401468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.401267 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.401468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.401327 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8649w\" (UniqueName: \"kubernetes.io/projected/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-kube-api-access-8649w\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.401468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.401384 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-metrics-certs\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.411380 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.411353 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-df7236-serving-cert\"" Nov 03 06:04:13.414120 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.414089 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5"] Nov 03 06:04:13.502439 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.502403 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.502722 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.502472 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.502722 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.502507 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8649w\" (UniqueName: \"kubernetes.io/projected/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-kube-api-access-8649w\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.502722 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.502545 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-metrics-certs\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.502722 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:04:13.502576 4855 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-serving-cert: secret "maas-api-e2e-worker-w0-df7236-serving-cert" not found Nov 03 06:04:13.502722 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:04:13.502643 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-maas-api-tls podName:5809e76e-9af0-4889-bf3c-5b9b7102eb1d nodeName:}" failed. No retries permitted until 2026-11-03 06:04:14.002622653 +0000 UTC m=+541.811009918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-maas-api-tls") pod "maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" (UID: "5809e76e-9af0-4889-bf3c-5b9b7102eb1d") : secret "maas-api-e2e-worker-w0-df7236-serving-cert" not found Nov 03 06:04:13.503840 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.503812 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.505080 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.505059 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-metrics-certs\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:13.530884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:13.530847 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8649w\" (UniqueName: \"kubernetes.io/projected/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-kube-api-access-8649w\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:14.008189 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:14.008152 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:14.010804 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:14.010778 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:14.291421 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:14.291321 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:14.592085 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:14.592058 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5"] Nov 03 06:04:14.596600 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:04:14.596565 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5809e76e_9af0_4889_bf3c_5b9b7102eb1d.slice/crio-b95a71e2d1aa5589c3f2705733e8c9ae642c42e96eab91e6f316357a5c6a467f WatchSource:0}: Error finding container b95a71e2d1aa5589c3f2705733e8c9ae642c42e96eab91e6f316357a5c6a467f: Status 404 returned error can't find the container with id b95a71e2d1aa5589c3f2705733e8c9ae642c42e96eab91e6f316357a5c6a467f Nov 03 06:04:14.623221 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:14.623181 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" event={"ID":"5809e76e-9af0-4889-bf3c-5b9b7102eb1d","Type":"ContainerStarted","Data":"b95a71e2d1aa5589c3f2705733e8c9ae642c42e96eab91e6f316357a5c6a467f"} Nov 03 06:04:15.540353 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.540320 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5"] Nov 03 06:04:15.544061 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.544042 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.563457 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.563436 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-5c3542-serving-cert\"" Nov 03 06:04:15.578616 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.578588 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5"] Nov 03 06:04:15.624475 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.624446 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d1643a-db61-492d-8b12-ef21e5eb0834-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.624910 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.624507 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmxsd\" (UniqueName: \"kubernetes.io/projected/34d1643a-db61-492d-8b12-ef21e5eb0834-kube-api-access-vmxsd\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.624910 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.624644 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-metrics-certs\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.624910 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.624795 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.630665 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.630636 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" event={"ID":"5809e76e-9af0-4889-bf3c-5b9b7102eb1d","Type":"ContainerStarted","Data":"7be8ecbbd209907c93a0ef4e24893a7ece5d1b946bc33c68d2c9a0b314e9031d"} Nov 03 06:04:15.630770 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.630729 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:15.679905 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.679854 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" podStartSLOduration=2.315972656 podStartE2EDuration="2.679838186s" totalImagesPullingTime="363.86553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:15.677716031 +0000 UTC m=+543.486103313" watchObservedRunningTime="2026-11-03 06:04:15.679838186 +0000 UTC m=+543.488225453" Nov 03 06:04:15.726240 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.726205 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.726444 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.726289 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d1643a-db61-492d-8b12-ef21e5eb0834-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.726444 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.726341 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmxsd\" (UniqueName: \"kubernetes.io/projected/34d1643a-db61-492d-8b12-ef21e5eb0834-kube-api-access-vmxsd\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.726444 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.726369 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-metrics-certs\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.727214 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.727193 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d1643a-db61-492d-8b12-ef21e5eb0834-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.728962 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.728941 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.729046 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.728980 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-metrics-certs\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.750945 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.750912 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmxsd\" (UniqueName: \"kubernetes.io/projected/34d1643a-db61-492d-8b12-ef21e5eb0834-kube-api-access-vmxsd\") pod \"maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:15.857753 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:15.857715 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:16.180627 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:16.180556 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5"] Nov 03 06:04:16.184017 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:04:16.183988 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d1643a_db61_492d_8b12_ef21e5eb0834.slice/crio-f9ea53686d3aa37f26a4a3f227fc5f6f6c729dc3eef9935444d0a9487e848d13 WatchSource:0}: Error finding container f9ea53686d3aa37f26a4a3f227fc5f6f6c729dc3eef9935444d0a9487e848d13: Status 404 returned error can't find the container with id f9ea53686d3aa37f26a4a3f227fc5f6f6c729dc3eef9935444d0a9487e848d13 Nov 03 06:04:16.637481 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:16.637445 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" event={"ID":"34d1643a-db61-492d-8b12-ef21e5eb0834","Type":"ContainerStarted","Data":"11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc"} Nov 03 06:04:16.637962 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:16.637502 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" event={"ID":"34d1643a-db61-492d-8b12-ef21e5eb0834","Type":"ContainerStarted","Data":"f9ea53686d3aa37f26a4a3f227fc5f6f6c729dc3eef9935444d0a9487e848d13"} Nov 03 06:04:16.637962 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:16.637718 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:16.683324 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:16.683260 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" podStartSLOduration=1.362983955 podStartE2EDuration="1.68324146s" totalImagesPullingTime="320.257505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:16.682115982 +0000 UTC m=+544.490503251" watchObservedRunningTime="2026-11-03 06:04:16.68324146 +0000 UTC m=+544.491628731" Nov 03 06:04:17.741699 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:17.741661 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:04:19.327791 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.327760 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6"] Nov 03 06:04:19.330908 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.330888 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:04:19.349519 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.349494 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6"] Nov 03 06:04:19.372964 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.372933 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-be1a66-dockercfg-vhsdx\"" Nov 03 06:04:19.380376 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.380345 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rtl\" (UniqueName: \"kubernetes.io/projected/97ba4903-d54a-4291-bd8b-811bc6b3302a-kube-api-access-82rtl\") pod \"payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6\" (UID: \"97ba4903-d54a-4291-bd8b-811bc6b3302a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:04:19.380530 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.380426 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ba4903-d54a-4291-bd8b-811bc6b3302a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6\" (UID: \"97ba4903-d54a-4291-bd8b-811bc6b3302a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:04:19.399554 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.399529 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-be1a66\"" Nov 03 06:04:19.482055 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.482012 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rtl\" (UniqueName: \"kubernetes.io/projected/97ba4903-d54a-4291-bd8b-811bc6b3302a-kube-api-access-82rtl\") pod \"payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6\" (UID: \"97ba4903-d54a-4291-bd8b-811bc6b3302a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:04:19.482223 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.482104 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ba4903-d54a-4291-bd8b-811bc6b3302a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6\" (UID: \"97ba4903-d54a-4291-bd8b-811bc6b3302a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:04:19.482800 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.482781 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ba4903-d54a-4291-bd8b-811bc6b3302a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6\" (UID: \"97ba4903-d54a-4291-bd8b-811bc6b3302a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:04:19.506287 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.506264 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rtl\" (UniqueName: \"kubernetes.io/projected/97ba4903-d54a-4291-bd8b-811bc6b3302a-kube-api-access-82rtl\") pod \"payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6\" (UID: \"97ba4903-d54a-4291-bd8b-811bc6b3302a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:04:19.644693 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.644598 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:04:19.948552 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:19.948529 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6"] Nov 03 06:04:19.951177 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:04:19.951149 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ba4903_d54a_4291_bd8b_811bc6b3302a.slice/crio-52a1543c17e2364901511c32ddd952af1ebc1cdf92a17aebc585ca329b2f16d0 WatchSource:0}: Error finding container 52a1543c17e2364901511c32ddd952af1ebc1cdf92a17aebc585ca329b2f16d0: Status 404 returned error can't find the container with id 52a1543c17e2364901511c32ddd952af1ebc1cdf92a17aebc585ca329b2f16d0 Nov 03 06:04:20.202094 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.202017 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm"] Nov 03 06:04:20.206998 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.206978 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:04:20.223201 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.223167 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b7e447\"" Nov 03 06:04:20.224513 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.224491 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm"] Nov 03 06:04:20.291105 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.291065 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536311c9-35e9-4309-a39a-aa592e4e3112-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm\" (UID: \"536311c9-35e9-4309-a39a-aa592e4e3112\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:04:20.291265 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.291175 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzps9\" (UniqueName: \"kubernetes.io/projected/536311c9-35e9-4309-a39a-aa592e4e3112-kube-api-access-rzps9\") pod \"payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm\" (UID: \"536311c9-35e9-4309-a39a-aa592e4e3112\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:04:20.392398 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.392239 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzps9\" (UniqueName: \"kubernetes.io/projected/536311c9-35e9-4309-a39a-aa592e4e3112-kube-api-access-rzps9\") pod \"payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm\" (UID: \"536311c9-35e9-4309-a39a-aa592e4e3112\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:04:20.393051 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.393016 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536311c9-35e9-4309-a39a-aa592e4e3112-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm\" (UID: \"536311c9-35e9-4309-a39a-aa592e4e3112\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:04:20.393984 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.393956 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536311c9-35e9-4309-a39a-aa592e4e3112-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm\" (UID: \"536311c9-35e9-4309-a39a-aa592e4e3112\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:04:20.417691 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.417660 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzps9\" (UniqueName: \"kubernetes.io/projected/536311c9-35e9-4309-a39a-aa592e4e3112-kube-api-access-rzps9\") pod \"payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm\" (UID: \"536311c9-35e9-4309-a39a-aa592e4e3112\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:04:20.521824 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.521739 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:04:20.658754 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.658721 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" event={"ID":"97ba4903-d54a-4291-bd8b-811bc6b3302a","Type":"ContainerStarted","Data":"52a1543c17e2364901511c32ddd952af1ebc1cdf92a17aebc585ca329b2f16d0"} Nov 03 06:04:20.845922 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.845849 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm"] Nov 03 06:04:20.848780 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:04:20.848754 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536311c9_35e9_4309_a39a_aa592e4e3112.slice/crio-5f2f0089a8a845dd5ba51aaac4ad5ab4fdbc29d84d52997e98e0ec891b2b3308 WatchSource:0}: Error finding container 5f2f0089a8a845dd5ba51aaac4ad5ab4fdbc29d84d52997e98e0ec891b2b3308: Status 404 returned error can't find the container with id 5f2f0089a8a845dd5ba51aaac4ad5ab4fdbc29d84d52997e98e0ec891b2b3308 Nov 03 06:04:20.962916 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.962879 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79"] Nov 03 06:04:20.966829 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.966804 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:20.985755 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.985728 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79"] Nov 03 06:04:20.986750 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:20.986728 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-ba6892-serving-cert\"" Nov 03 06:04:21.100204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.100121 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.100344 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.100245 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-metrics-certs\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.100344 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.100308 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b12a4c2c-993c-448b-b051-c68fa6dc4733-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.100479 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.100440 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9x64\" (UniqueName: \"kubernetes.io/projected/b12a4c2c-993c-448b-b051-c68fa6dc4733-kube-api-access-w9x64\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.201333 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.201286 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9x64\" (UniqueName: \"kubernetes.io/projected/b12a4c2c-993c-448b-b051-c68fa6dc4733-kube-api-access-w9x64\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.201971 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.201940 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.202542 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.202261 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-metrics-certs\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.202542 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.202304 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b12a4c2c-993c-448b-b051-c68fa6dc4733-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.203231 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.203205 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b12a4c2c-993c-448b-b051-c68fa6dc4733-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.216214 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.216180 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.216819 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.216795 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-metrics-certs\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.226990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.226964 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9x64\" (UniqueName: \"kubernetes.io/projected/b12a4c2c-993c-448b-b051-c68fa6dc4733-kube-api-access-w9x64\") pod \"maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.281503 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.281470 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:21.614030 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.613949 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79"] Nov 03 06:04:21.627968 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:04:21.618511 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12a4c2c_993c_448b_b051_c68fa6dc4733.slice/crio-65c0b7dd29bc0b643daffc88d14f997610f9463b8dbd151acee661b078358469 WatchSource:0}: Error finding container 65c0b7dd29bc0b643daffc88d14f997610f9463b8dbd151acee661b078358469: Status 404 returned error can't find the container with id 65c0b7dd29bc0b643daffc88d14f997610f9463b8dbd151acee661b078358469 Nov 03 06:04:21.644173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.644151 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:04:21.666742 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.666704 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" event={"ID":"536311c9-35e9-4309-a39a-aa592e4e3112","Type":"ContainerStarted","Data":"5f2f0089a8a845dd5ba51aaac4ad5ab4fdbc29d84d52997e98e0ec891b2b3308"} Nov 03 06:04:21.668478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:21.668376 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" event={"ID":"b12a4c2c-993c-448b-b051-c68fa6dc4733","Type":"ContainerStarted","Data":"65c0b7dd29bc0b643daffc88d14f997610f9463b8dbd151acee661b078358469"} Nov 03 06:04:22.648400 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:22.648357 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:04:22.675650 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:22.675613 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" event={"ID":"536311c9-35e9-4309-a39a-aa592e4e3112","Type":"ContainerStarted","Data":"acda9f71ce6486797dcdc558701e986441597807a6a587696c7c3ce458cfb428"} Nov 03 06:04:22.675821 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:22.675717 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:04:22.677218 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:22.677192 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" event={"ID":"b12a4c2c-993c-448b-b051-c68fa6dc4733","Type":"ContainerStarted","Data":"bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198"} Nov 03 06:04:22.677358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:22.677239 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:22.678827 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:22.678807 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" event={"ID":"97ba4903-d54a-4291-bd8b-811bc6b3302a","Type":"ContainerStarted","Data":"e4f569de32d758da29731687c86c635b3a3f96c201349090cf278d77c08deaea"} Nov 03 06:04:22.678951 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:22.678936 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:04:22.781984 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:22.781935 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" podStartSLOduration=1.442090723 podStartE2EDuration="3.78191975s" totalImagesPullingTime="2.339829027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:22.781215054 +0000 UTC m=+550.589602346" watchObservedRunningTime="2026-11-03 06:04:22.78191975 +0000 UTC m=+550.590307017" Nov 03 06:04:22.829873 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:22.829813 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" podStartSLOduration=1.3017079360000001 podStartE2EDuration="2.829796115s" totalImagesPullingTime="1.528088179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:22.826371884 +0000 UTC m=+550.634759148" watchObservedRunningTime="2026-11-03 06:04:22.829796115 +0000 UTC m=+550.638183383" Nov 03 06:04:22.872722 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:22.872616 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" podStartSLOduration=2.524219437 podStartE2EDuration="2.872602039s" totalImagesPullingTime="348.382602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:22.872192452 +0000 UTC m=+550.680579720" watchObservedRunningTime="2026-11-03 06:04:22.872602039 +0000 UTC m=+550.680989307" Nov 03 06:04:23.551961 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:23.551918 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:04:28.489562 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.489523 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b"] Nov 03 06:04:28.493368 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.493347 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.513358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.513240 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-86728b-serving-cert\"" Nov 03 06:04:28.513751 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.513717 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b"] Nov 03 06:04:28.620918 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.620877 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-metrics-certs\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.621081 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.620999 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea38a440-6197-4e41-b316-955e76211942-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.621126 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.621112 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45qz\" (UniqueName: \"kubernetes.io/projected/ea38a440-6197-4e41-b316-955e76211942-kube-api-access-h45qz\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.621165 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.621151 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-maas-api-tls\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.689672 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.689643 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:04:28.721888 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.721842 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45qz\" (UniqueName: \"kubernetes.io/projected/ea38a440-6197-4e41-b316-955e76211942-kube-api-access-h45qz\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.721888 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.721900 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-maas-api-tls\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.722133 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.721940 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-metrics-certs\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.722133 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.721986 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea38a440-6197-4e41-b316-955e76211942-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.722780 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.722754 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea38a440-6197-4e41-b316-955e76211942-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.724523 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.724498 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-maas-api-tls\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.724624 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.724602 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-metrics-certs\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.764295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.764222 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45qz\" (UniqueName: \"kubernetes.io/projected/ea38a440-6197-4e41-b316-955e76211942-kube-api-access-h45qz\") pod \"maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.806198 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:28.806162 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:28.919154 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:04:28.919085 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:04:29.159566 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:29.159537 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b"] Nov 03 06:04:29.162626 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:04:29.162591 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea38a440_6197_4e41_b316_955e76211942.slice/crio-28a967dd6345ee7c88bdf5f7c22b9d5f56e9bc2d3ff2ffbb89f7292b75947ffa WatchSource:0}: Error finding container 28a967dd6345ee7c88bdf5f7c22b9d5f56e9bc2d3ff2ffbb89f7292b75947ffa: Status 404 returned error can't find the container with id 28a967dd6345ee7c88bdf5f7c22b9d5f56e9bc2d3ff2ffbb89f7292b75947ffa Nov 03 06:04:29.723830 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:29.723796 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" event={"ID":"ea38a440-6197-4e41-b316-955e76211942","Type":"ContainerStarted","Data":"28a967dd6345ee7c88bdf5f7c22b9d5f56e9bc2d3ff2ffbb89f7292b75947ffa"} Nov 03 06:04:30.358765 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:30.358721 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:04:30.731108 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:30.731021 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" event={"ID":"ea38a440-6197-4e41-b316-955e76211942","Type":"ContainerStarted","Data":"5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c"} Nov 03 06:04:30.731518 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:30.731158 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:30.782108 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:30.782056 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" podStartSLOduration=2.098678659 podStartE2EDuration="2.782043152s" totalImagesPullingTime="683.364493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:30.780715902 +0000 UTC m=+558.589103173" watchObservedRunningTime="2026-11-03 06:04:30.782043152 +0000 UTC m=+558.590430419" Nov 03 06:04:33.688168 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:33.688137 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:04:33.688674 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:33.688300 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:04:35.817832 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:35.817801 4855 generic.go:356] "Generic (PLEG): container finished" podID="5507309e-a297-480d-bd20-4d3914fe9cae" containerID="5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53" exitCode=0 Nov 03 06:04:35.818273 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:35.817878 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" event={"ID":"5507309e-a297-480d-bd20-4d3914fe9cae","Type":"ContainerDied","Data":"5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53"} Nov 03 06:04:35.818506 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:35.818490 4855 scope.go:122] "RemoveContainer" containerID="5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53" Nov 03 06:04:35.820352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:35.820003 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:35.820352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:35.820067 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:35.820352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:35.820098 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:36.788422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:36.785058 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:04:36.826819 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:36.826782 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" event={"ID":"5507309e-a297-480d-bd20-4d3914fe9cae","Type":"ContainerStarted","Data":"6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e"} Nov 03 06:04:37.577485 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:37.577452 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:37.578799 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:37.578773 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 06:04:37.578943 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:37.578826 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 06:04:38.577699 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:38.577663 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 06:04:38.578099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:38.577717 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 06:04:39.220038 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:39.219997 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:04:39.578026 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:39.577989 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 06:04:39.578447 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:39.578054 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 06:04:40.577617 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:40.577575 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 06:04:40.577796 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:40.577653 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 06:04:40.595482 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:40.595428 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:41.186160 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.184329 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": read tcp 10.133.0.2:43448->10.133.0.40:15021: read: connection reset by peer" start-of-body= Nov 03 06:04:41.186160 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.184421 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": read tcp 10.133.0.2:43448->10.133.0.40:15021: read: connection reset by peer" Nov 03 06:04:41.578127 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.578095 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 06:04:41.578292 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.578145 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 06:04:41.604937 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.604905 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 06:04:41.605278 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.604951 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 06:04:41.889738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.889651 4855 generic.go:356] "Generic (PLEG): container finished" podID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerID="4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218" exitCode=0 Nov 03 06:04:41.889738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.889723 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" event={"ID":"d926084c-dafb-4fd7-adc3-928dc9036a92","Type":"ContainerDied","Data":"4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218"} Nov 03 06:04:41.890173 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.890155 4855 scope.go:122] "RemoveContainer" containerID="4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218" Nov 03 06:04:41.891512 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.891484 4855 generic.go:356] "Generic (PLEG): container finished" podID="299590cc-de3d-4408-8653-320887002525" containerID="6bafdf370835679a5d1fa10ac864ba0fef9f5d1933cec2f0e1c95b6c7d86e03b" exitCode=0 Nov 03 06:04:41.891609 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.891526 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" event={"ID":"299590cc-de3d-4408-8653-320887002525","Type":"ContainerDied","Data":"6bafdf370835679a5d1fa10ac864ba0fef9f5d1933cec2f0e1c95b6c7d86e03b"} Nov 03 06:04:41.891609 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.891522 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:41.891682 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.891647 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:41.891726 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.891679 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:41.891819 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.891806 4855 scope.go:122] "RemoveContainer" containerID="6bafdf370835679a5d1fa10ac864ba0fef9f5d1933cec2f0e1c95b6c7d86e03b" Nov 03 06:04:41.892947 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.892911 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:41.893030 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.892992 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:41.893082 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:41.893026 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:04:42.734803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:42.734773 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:42.750425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:42.747663 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:04:42.907745 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:42.907711 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" event={"ID":"d926084c-dafb-4fd7-adc3-928dc9036a92","Type":"ContainerStarted","Data":"917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b"} Nov 03 06:04:42.910472 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:42.910448 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" event={"ID":"299590cc-de3d-4408-8653-320887002525","Type":"ContainerStarted","Data":"f76325e9988ab6f7b56ba91e4de6be40b61567e73c751ef405c6621f27442604"} Nov 03 06:04:43.071415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:43.062451 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:43.071415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:43.062655 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 06:04:43.071415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:43.062695 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 06:04:43.736736 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:43.736704 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:43.737483 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:43.737457 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 06:04:43.737678 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:43.737653 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 06:04:44.045474 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:44.042774 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 06:04:44.045474 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:44.042836 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 06:04:44.738810 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:44.738770 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 06:04:44.739410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:44.739363 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 06:04:45.050476 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:45.040790 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 06:04:45.050476 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:45.040854 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 06:04:45.737965 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:45.737909 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 06:04:45.738283 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:45.738250 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 06:04:46.041090 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:46.040975 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 06:04:46.041775 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:46.041728 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 06:04:46.742415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:46.738795 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 06:04:46.742415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:46.738870 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 06:04:47.043916 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:47.043821 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 06:04:47.044529 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:47.044486 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 06:04:47.740878 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:47.740839 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 06:04:47.741181 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:47.741147 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 06:04:48.041206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:48.041124 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 06:04:48.041352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:48.041196 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 06:04:48.741019 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:48.740982 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:48.741435 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:48.741299 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:48.742004 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:48.741986 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:04:49.045362 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:49.045290 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:49.045686 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:49.045670 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:49.046466 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:49.046452 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:04:51.237960 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.237926 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2"] Nov 03 06:04:51.243955 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.243926 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.260210 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.260187 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2"] Nov 03 06:04:51.272066 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.272037 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-be1a66\"/\"openshift-service-ca.crt\"" Nov 03 06:04:51.272066 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.272064 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-be1a66\"/\"kube-root-ca.crt\"" Nov 03 06:04:51.272291 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.272124 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be1a66\"/\"default-dockercfg-xtbd8\"" Nov 03 06:04:51.275155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.275139 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be1a66\"/\"simulator-w4-be1a66-kserve-self-signed-certs\"" Nov 03 06:04:51.328852 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.328824 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7qzj\" (UniqueName: \"kubernetes.io/projected/9eea1294-b376-4523-8b28-5a213d66596c-kube-api-access-b7qzj\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.329037 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.328860 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-tmp-dir\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.329037 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.328883 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-model-cache\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.329037 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.328988 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eea1294-b376-4523-8b28-5a213d66596c-tls-certs\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.329037 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.329012 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-dshm\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.329207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.329048 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-home\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.430301 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.430267 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7qzj\" (UniqueName: \"kubernetes.io/projected/9eea1294-b376-4523-8b28-5a213d66596c-kube-api-access-b7qzj\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.430512 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.430311 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-tmp-dir\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.430512 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.430337 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-model-cache\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.430512 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.430370 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eea1294-b376-4523-8b28-5a213d66596c-tls-certs\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.430512 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.430415 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-dshm\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.430512 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.430474 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-home\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.430883 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.430857 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-model-cache\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.430945 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.430893 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-tmp-dir\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.430983 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.430938 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-home\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.432712 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.432692 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-dshm\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.433043 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.433028 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eea1294-b376-4523-8b28-5a213d66596c-tls-certs\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.455338 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.455303 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7qzj\" (UniqueName: \"kubernetes.io/projected/9eea1294-b376-4523-8b28-5a213d66596c-kube-api-access-b7qzj\") pod \"simulator-w4-be1a66-kserve-7fdd59dd46-928z2\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.558616 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.558581 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:51.763020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.762984 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:04:51.861012 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.860937 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2"] Nov 03 06:04:51.864163 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:04:51.864132 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eea1294_b376_4523_8b28_5a213d66596c.slice/crio-ac40ec62af0329e9cf9bd2c7c1d6626d9514f8f19f130c29a7d145319685dac3 WatchSource:0}: Error finding container ac40ec62af0329e9cf9bd2c7c1d6626d9514f8f19f130c29a7d145319685dac3: Status 404 returned error can't find the container with id ac40ec62af0329e9cf9bd2c7c1d6626d9514f8f19f130c29a7d145319685dac3 Nov 03 06:04:51.983323 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.983289 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" event={"ID":"9eea1294-b376-4523-8b28-5a213d66596c","Type":"ContainerStarted","Data":"7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3"} Nov 03 06:04:51.983498 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:51.983332 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" event={"ID":"9eea1294-b376-4523-8b28-5a213d66596c","Type":"ContainerStarted","Data":"ac40ec62af0329e9cf9bd2c7c1d6626d9514f8f19f130c29a7d145319685dac3"} Nov 03 06:04:52.027795 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:52.027744 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podStartSLOduration=1.027729781 podStartE2EDuration="1.027729781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:52.0266219 +0000 UTC m=+579.835009162" watchObservedRunningTime="2026-11-03 06:04:52.027729781 +0000 UTC m=+579.836117073" Nov 03 06:04:52.559212 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:04:52.559176 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:04:58.883902 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:04:58.883829 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:05:01.559671 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:01.559628 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:05:01.572502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:01.572467 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:05:02.049059 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:02.049034 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:05:04.268354 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:04.268319 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:05:12.382582 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:12.382555 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:05:12.383861 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:12.383841 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:05:16.557401 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:16.557353 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:05:17.857598 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:17.857559 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf"] Nov 03 06:05:17.863040 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:17.863016 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:17.880092 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:17.880068 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf"] Nov 03 06:05:17.891632 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:17.891601 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5c3542\"/\"kube-root-ca.crt\"" Nov 03 06:05:17.892099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:17.892076 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5c3542\"/\"premium-w2-5c3542-kserve-self-signed-certs\"" Nov 03 06:05:17.892099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:17.892077 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5c3542\"/\"default-dockercfg-nhbrc\"" Nov 03 06:05:17.892251 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:17.892083 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5c3542\"/\"openshift-service-ca.crt\"" Nov 03 06:05:18.018333 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.018300 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e65c31d-5628-4811-a84e-405aa9eb9188-tls-certs\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.018522 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.018347 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-model-cache\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.018522 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.018369 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-home\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.018522 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.018484 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-dshm\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.018645 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.018538 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nz7h\" (UniqueName: \"kubernetes.io/projected/8e65c31d-5628-4811-a84e-405aa9eb9188-kube-api-access-2nz7h\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.018645 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.018561 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-tmp-dir\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.119872 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.119790 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-home\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.119872 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.119825 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-dshm\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.119872 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.119849 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nz7h\" (UniqueName: \"kubernetes.io/projected/8e65c31d-5628-4811-a84e-405aa9eb9188-kube-api-access-2nz7h\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.120095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.119880 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-tmp-dir\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.120095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.120011 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e65c31d-5628-4811-a84e-405aa9eb9188-tls-certs\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.120095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.120056 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-model-cache\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.120265 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.120239 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-home\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.120336 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.120320 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-tmp-dir\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.120408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.120368 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-model-cache\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.122138 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.122105 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-dshm\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.122416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.122383 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e65c31d-5628-4811-a84e-405aa9eb9188-tls-certs\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.145765 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.145742 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nz7h\" (UniqueName: \"kubernetes.io/projected/8e65c31d-5628-4811-a84e-405aa9eb9188-kube-api-access-2nz7h\") pod \"premium-w2-5c3542-kserve-6666fb747d-9vwwf\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.176325 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.176298 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:18.478935 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:18.478908 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf"] Nov 03 06:05:18.481317 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:05:18.481281 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e65c31d_5628_4811_a84e_405aa9eb9188.slice/crio-f6d9b0874ef2d39f33dce7d09516fa1c9c4134d896985a67fe80d8528fe50e19 WatchSource:0}: Error finding container f6d9b0874ef2d39f33dce7d09516fa1c9c4134d896985a67fe80d8528fe50e19: Status 404 returned error can't find the container with id f6d9b0874ef2d39f33dce7d09516fa1c9c4134d896985a67fe80d8528fe50e19 Nov 03 06:05:19.135881 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:19.135838 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" event={"ID":"8e65c31d-5628-4811-a84e-405aa9eb9188","Type":"ContainerStarted","Data":"afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc"} Nov 03 06:05:19.136254 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:19.135891 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" event={"ID":"8e65c31d-5628-4811-a84e-405aa9eb9188","Type":"ContainerStarted","Data":"f6d9b0874ef2d39f33dce7d09516fa1c9c4134d896985a67fe80d8528fe50e19"} Nov 03 06:05:19.176535 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:19.176506 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:19.180773 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:19.180729 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podStartSLOduration=2.180715256 podStartE2EDuration="2.180715256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:05:19.178921461 +0000 UTC m=+606.987308729" watchObservedRunningTime="2026-11-03 06:05:19.180715256 +0000 UTC m=+606.989102524" Nov 03 06:05:28.176989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:28.176949 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:28.189961 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:28.189935 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:28.883828 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:05:28.883778 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:05:29.189561 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:29.189489 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:05:34.148098 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:34.148056 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:05:44.076123 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.076087 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv"] Nov 03 06:05:44.080370 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.080345 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.099361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.099310 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ba6892\"/\"kube-root-ca.crt\"" Nov 03 06:05:44.112435 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.112408 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ba6892\"/\"openshift-service-ca.crt\"" Nov 03 06:05:44.112592 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.112541 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ba6892\"/\"default-dockercfg-v45nf\"" Nov 03 06:05:44.112687 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.112673 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ba6892\"/\"premium-w5-ba6892-kserve-self-signed-certs\"" Nov 03 06:05:44.114938 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.114917 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv"] Nov 03 06:05:44.200951 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.200921 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-home\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.201107 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.200958 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjw28\" (UniqueName: \"kubernetes.io/projected/f485f34b-569a-4647-b63f-3b1ce1f5b2df-kube-api-access-pjw28\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.201107 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.200989 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-dshm\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.201107 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.201044 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tmp-dir\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.201209 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.201179 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tls-certs\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.201288 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.201247 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-model-cache\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.302184 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.302144 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-model-cache\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.302367 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.302244 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-home\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.302367 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.302285 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjw28\" (UniqueName: \"kubernetes.io/projected/f485f34b-569a-4647-b63f-3b1ce1f5b2df-kube-api-access-pjw28\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.302367 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.302331 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-dshm\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.302367 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.302353 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tmp-dir\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.302596 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.302434 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tls-certs\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.303498 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.303448 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-home\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.303617 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.303538 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-model-cache\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.303669 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.303655 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tmp-dir\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.307943 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.307918 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-dshm\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.308162 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.308145 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tls-certs\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.327939 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.327868 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjw28\" (UniqueName: \"kubernetes.io/projected/f485f34b-569a-4647-b63f-3b1ce1f5b2df-kube-api-access-pjw28\") pod \"premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.395252 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.395214 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:44.697017 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:44.696990 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv"] Nov 03 06:05:44.699073 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:05:44.699046 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf485f34b_569a_4647_b63f_3b1ce1f5b2df.slice/crio-a6ba590128ed03de47501796a9e3bb42715b7a8c1a6733498c92814e0d4448a3 WatchSource:0}: Error finding container a6ba590128ed03de47501796a9e3bb42715b7a8c1a6733498c92814e0d4448a3: Status 404 returned error can't find the container with id a6ba590128ed03de47501796a9e3bb42715b7a8c1a6733498c92814e0d4448a3 Nov 03 06:05:45.289126 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:45.289085 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" event={"ID":"f485f34b-569a-4647-b63f-3b1ce1f5b2df","Type":"ContainerStarted","Data":"b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040"} Nov 03 06:05:45.289865 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:45.289135 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" event={"ID":"f485f34b-569a-4647-b63f-3b1ce1f5b2df","Type":"ContainerStarted","Data":"a6ba590128ed03de47501796a9e3bb42715b7a8c1a6733498c92814e0d4448a3"} Nov 03 06:05:45.333696 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:45.333646 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podStartSLOduration=1.333631304 podStartE2EDuration="1.333631304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:05:45.332271321 +0000 UTC m=+633.140658580" watchObservedRunningTime="2026-11-03 06:05:45.333631304 +0000 UTC m=+633.142018571" Nov 03 06:05:45.396144 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:45.396118 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:53.299455 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.299422 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk"] Nov 03 06:05:53.304398 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.304364 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.320629 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.320605 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5c3542-openshift-default-dockercfg-mhnlv\"" Nov 03 06:05:53.336174 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.336148 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk"] Nov 03 06:05:53.414293 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.414251 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.414486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.414316 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a24180-c194-4ead-810e-5e20c52cfbb9-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.414486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.414427 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.414486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.414457 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.414618 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.414496 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-945ph\" (UniqueName: \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-kube-api-access-945ph\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.414618 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.414554 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.414618 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.414577 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.414618 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.414614 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.414806 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.414671 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.515739 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.515705 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.515739 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.515740 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.515960 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.515798 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.515960 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.515824 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.515960 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.515879 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.515960 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.515929 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a24180-c194-4ead-810e-5e20c52cfbb9-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.516174 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.515979 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.516174 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.516012 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.516174 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.516050 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-945ph\" (UniqueName: \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-kube-api-access-945ph\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.516174 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.516162 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.516376 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.516280 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.516376 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.516331 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.516504 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.516429 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.516791 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.516771 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a24180-c194-4ead-810e-5e20c52cfbb9-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.518219 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.518196 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.518469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.518451 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.540322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.540293 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-945ph\" (UniqueName: \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-kube-api-access-945ph\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.540435 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.540299 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.618049 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.617969 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:53.936286 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.936248 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk"] Nov 03 06:05:53.938651 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:05:53.938623 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a24180_c194_4ead_810e_5e20c52cfbb9.slice/crio-db5d1b4cbe1b774bc7a52f0483d0fbfd7aee4bd09cad55a464bc5e22f8ebd8e9 WatchSource:0}: Error finding container db5d1b4cbe1b774bc7a52f0483d0fbfd7aee4bd09cad55a464bc5e22f8ebd8e9: Status 404 returned error can't find the container with id db5d1b4cbe1b774bc7a52f0483d0fbfd7aee4bd09cad55a464bc5e22f8ebd8e9 Nov 03 06:05:53.940891 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.940859 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:05:53.940961 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.940924 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:05:53.940998 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:53.940959 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:05:54.382217 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:54.382178 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" event={"ID":"c2a24180-c194-4ead-810e-5e20c52cfbb9","Type":"ContainerStarted","Data":"7e0e80823f4ba203a79b6527166f92c61ec0449913fb8cea2de4587069cc6abc"} Nov 03 06:05:54.382809 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:54.382783 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" event={"ID":"c2a24180-c194-4ead-810e-5e20c52cfbb9","Type":"ContainerStarted","Data":"db5d1b4cbe1b774bc7a52f0483d0fbfd7aee4bd09cad55a464bc5e22f8ebd8e9"} Nov 03 06:05:54.396509 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:54.396476 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:54.414807 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:54.414774 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:54.452280 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:54.450881 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" podStartSLOduration=1.450861457 podStartE2EDuration="1.450861457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:05:54.448553362 +0000 UTC m=+642.256940630" watchObservedRunningTime="2026-11-03 06:05:54.450861457 +0000 UTC m=+642.259248727" Nov 03 06:05:54.618787 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:54.618756 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:05:54.620581 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:54.620550 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 06:05:54.620696 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:54.620597 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 06:05:55.403034 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:55.403008 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:05:55.619005 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:55.618967 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 06:05:55.619189 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:55.619030 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 06:05:56.619218 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:56.619181 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 06:05:56.619666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:56.619236 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 06:05:57.619366 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:57.619331 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 06:05:57.619792 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:57.619385 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 06:05:58.618531 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:58.618498 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 06:05:58.618712 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:58.618551 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 06:05:58.883479 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:05:58.883373 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:05:59.618725 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:59.618688 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 06:05:59.618913 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:05:59.618745 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 06:06:00.623893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:00.623858 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:06:00.624334 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:00.624179 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:06:00.624961 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:00.624943 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:06:00.750793 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:00.750761 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:06:24.065174 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.064971 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp"] Nov 03 06:06:24.074946 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.074914 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.090442 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.090410 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp"] Nov 03 06:06:24.102203 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.102157 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-86728b-openshift-default-dockercfg-kswxd\"" Nov 03 06:06:24.187651 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.187608 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee92591f-8473-4981-8af8-bb60f096c653-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.187830 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.187659 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.187830 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.187717 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.187830 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.187753 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.187830 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.187777 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z9dt\" (UniqueName: \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-kube-api-access-5z9dt\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.187997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.187861 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.187997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.187928 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.187997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.187952 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee92591f-8473-4981-8af8-bb60f096c653-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.187997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.187973 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.288732 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.288692 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.288732 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.288737 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee92591f-8473-4981-8af8-bb60f096c653-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.288977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.288764 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.288977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.288879 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee92591f-8473-4981-8af8-bb60f096c653-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.288977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.288927 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.288977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.288954 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.289240 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.289007 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.289240 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.289093 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.289240 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.289127 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z9dt\" (UniqueName: \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-kube-api-access-5z9dt\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.289240 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.289177 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.289487 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.289270 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.289487 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.289341 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.289598 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.289555 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.289598 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.289560 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee92591f-8473-4981-8af8-bb60f096c653-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.291464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.291441 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee92591f-8473-4981-8af8-bb60f096c653-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.291464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.291443 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.313759 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.313726 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.313937 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.313916 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z9dt\" (UniqueName: \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-kube-api-access-5z9dt\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.390454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.390343 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:24.711296 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.711271 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp"] Nov 03 06:06:24.714529 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:06:24.714499 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee92591f_8473_4981_8af8_bb60f096c653.slice/crio-7c9527be9866fdfc4c423647ff623f38874eee25a662f2548d425518bd1c78d6 WatchSource:0}: Error finding container 7c9527be9866fdfc4c423647ff623f38874eee25a662f2548d425518bd1c78d6: Status 404 returned error can't find the container with id 7c9527be9866fdfc4c423647ff623f38874eee25a662f2548d425518bd1c78d6 Nov 03 06:06:24.716851 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.716818 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:06:24.716946 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.716886 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:06:24.716946 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:24.716919 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:06:25.563457 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:25.563382 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" event={"ID":"ee92591f-8473-4981-8af8-bb60f096c653","Type":"ContainerStarted","Data":"244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5"} Nov 03 06:06:25.563990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:25.563480 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" event={"ID":"ee92591f-8473-4981-8af8-bb60f096c653","Type":"ContainerStarted","Data":"7c9527be9866fdfc4c423647ff623f38874eee25a662f2548d425518bd1c78d6"} Nov 03 06:06:25.628455 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:25.628403 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" podStartSLOduration=1.628373842 podStartE2EDuration="1.628373842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:25.627219373 +0000 UTC m=+673.435606642" watchObservedRunningTime="2026-11-03 06:06:25.628373842 +0000 UTC m=+673.436761110" Nov 03 06:06:26.390608 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:26.390577 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:26.392467 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:26.392433 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 06:06:26.392622 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:26.392495 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 06:06:27.391122 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:27.391079 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 06:06:27.391676 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:27.391177 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 06:06:28.391298 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:28.391269 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 06:06:28.391733 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:28.391327 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 06:06:28.885494 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:06:28.885457 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:06:29.391350 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:29.391319 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 06:06:29.391774 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:29.391379 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 06:06:30.391095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:30.391061 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 06:06:30.391279 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:30.391120 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 06:06:31.391015 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:31.390982 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 06:06:31.391445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:31.391034 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 06:06:32.395020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:32.394988 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:32.395495 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:32.395272 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:32.396177 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:32.396160 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:06:38.572535 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.572501 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb"] Nov 03 06:06:38.577294 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.577264 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.595561 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.595538 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb"] Nov 03 06:06:38.607870 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.606464 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-df7236-openshift-default-dockercfg-qtf5v\"" Nov 03 06:06:38.670282 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.670243 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3a0b0e-10e5-4605-aa0e-67745585646d-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.670496 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.670308 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.670496 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.670471 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.670496 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.670495 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.670657 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.670511 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.670657 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.670629 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.670738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.670662 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmsjx\" (UniqueName: \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-kube-api-access-mmsjx\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.670738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.670688 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.670797 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.670773 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772201 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772166 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772201 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772204 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772482 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772220 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772482 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772266 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772482 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772287 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmsjx\" (UniqueName: \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-kube-api-access-mmsjx\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772482 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772309 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772482 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772331 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772482 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772399 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3a0b0e-10e5-4605-aa0e-67745585646d-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772482 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772476 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772899 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772667 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772899 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772738 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.772899 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772789 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.773053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.772990 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.773120 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.773099 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3a0b0e-10e5-4605-aa0e-67745585646d-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.774765 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.774740 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.774929 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.774910 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.796874 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.796845 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.797064 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.797045 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmsjx\" (UniqueName: \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-kube-api-access-mmsjx\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:38.822696 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.822639 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:06:38.892301 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:38.892272 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:39.210696 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.210674 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb"] Nov 03 06:06:39.212449 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:06:39.212421 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3a0b0e_10e5_4605_aa0e_67745585646d.slice/crio-5af2190cbfb5c7e4afe8b241c72657f105c1653a7ac4d08f7a3ca178ccbcadb9 WatchSource:0}: Error finding container 5af2190cbfb5c7e4afe8b241c72657f105c1653a7ac4d08f7a3ca178ccbcadb9: Status 404 returned error can't find the container with id 5af2190cbfb5c7e4afe8b241c72657f105c1653a7ac4d08f7a3ca178ccbcadb9 Nov 03 06:06:39.214842 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.214811 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:06:39.214930 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.214876 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:06:39.214930 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.214911 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:06:39.426084 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.426049 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6"] Nov 03 06:06:39.429906 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.429885 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.450156 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.450130 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b7e447-openshift-default-dockercfg-slpt8\"" Nov 03 06:06:39.453956 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.453926 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6"] Nov 03 06:06:39.478603 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.478522 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.478740 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.478717 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slxw5\" (UniqueName: \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-kube-api-access-slxw5\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.478803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.478760 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.478893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.478871 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.478930 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.478917 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.478970 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.478946 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.479054 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.479039 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.479111 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.479095 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.479168 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.479136 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.580758 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.580720 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.580796 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.580826 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.580868 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.580989 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxw5\" (UniqueName: \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-kube-api-access-slxw5\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.581021 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.581068 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.581102 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.581127 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.581161 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.581310 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.581584 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.581790 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.581944 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.581800 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.584024 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.584000 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.584251 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.584225 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.606100 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.606018 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.606100 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.606101 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxw5\" (UniqueName: \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-kube-api-access-slxw5\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.681409 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.680887 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" event={"ID":"ec3a0b0e-10e5-4605-aa0e-67745585646d","Type":"ContainerStarted","Data":"8c9ac4e3dd1e867cc099dbdd318db6729f0fd37ee3058499c513675eeeb5e878"} Nov 03 06:06:39.681409 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.680957 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" event={"ID":"ec3a0b0e-10e5-4605-aa0e-67745585646d","Type":"ContainerStarted","Data":"5af2190cbfb5c7e4afe8b241c72657f105c1653a7ac4d08f7a3ca178ccbcadb9"} Nov 03 06:06:39.739685 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.739559 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podStartSLOduration=1.7395378460000002 podStartE2EDuration="1.739537846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:39.737906144 +0000 UTC m=+687.546293412" watchObservedRunningTime="2026-11-03 06:06:39.739537846 +0000 UTC m=+687.547925116" Nov 03 06:06:39.750887 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.750600 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:39.892854 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:39.892772 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:40.146449 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:40.145575 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6"] Nov 03 06:06:40.181435 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:06:40.175051 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb9581f_e4cd_4cbd_b20e_4beeff24063c.slice/crio-913fd7c12db4d3101c340f3d7ba9fffb72d688bd257012cbcf12a3c7ab321580 WatchSource:0}: Error finding container 913fd7c12db4d3101c340f3d7ba9fffb72d688bd257012cbcf12a3c7ab321580: Status 404 returned error can't find the container with id 913fd7c12db4d3101c340f3d7ba9fffb72d688bd257012cbcf12a3c7ab321580 Nov 03 06:06:40.194429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:40.193110 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:06:40.194429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:40.193242 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:06:40.194429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:40.193305 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:06:40.690429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:40.690313 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" event={"ID":"beb9581f-e4cd-4cbd-b20e-4beeff24063c","Type":"ContainerStarted","Data":"734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf"} Nov 03 06:06:40.691007 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:40.690978 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" event={"ID":"beb9581f-e4cd-4cbd-b20e-4beeff24063c","Type":"ContainerStarted","Data":"913fd7c12db4d3101c340f3d7ba9fffb72d688bd257012cbcf12a3c7ab321580"} Nov 03 06:06:40.741772 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:40.741703 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" podStartSLOduration=1.741685165 podStartE2EDuration="1.741685165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:40.74082978 +0000 UTC m=+688.549217049" watchObservedRunningTime="2026-11-03 06:06:40.741685165 +0000 UTC m=+688.550072433" Nov 03 06:06:40.754509 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:40.754474 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:40.894606 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:40.894572 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 06:06:40.894803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:40.894623 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 06:06:41.753414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:41.753356 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 06:06:41.753963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:41.753443 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 06:06:41.893183 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:41.893146 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 06:06:41.893448 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:41.893421 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 06:06:42.763896 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:42.762932 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 06:06:42.763896 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:42.763498 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 06:06:42.892876 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:42.892840 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 06:06:42.893040 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:42.892905 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 06:06:43.754830 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:43.754793 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 06:06:43.755132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:43.755103 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 06:06:43.893027 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:43.892997 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 06:06:43.893519 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:43.893491 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 06:06:44.755801 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:44.755761 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 06:06:44.756272 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:44.756237 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 06:06:44.892625 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:44.892582 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 06:06:44.892826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:44.892644 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 06:06:45.753832 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:45.753784 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 06:06:45.754357 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:45.753866 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 06:06:45.893142 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:45.893110 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 06:06:45.893307 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:45.893168 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 06:06:46.751578 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:46.751536 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 06:06:46.751766 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:46.751615 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 06:06:46.902076 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:46.902043 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:46.902576 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:46.902489 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:46.903231 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:46.903207 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:06:47.756450 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:47.756424 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:47.756687 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:47.756669 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:47.757257 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:06:47.757240 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:06:58.884220 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:06:58.884177 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:07:01.562640 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:01.562604 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:07:26.584470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.584433 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k"] Nov 03 06:07:26.589935 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.589904 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.610465 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.610432 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k"] Nov 03 06:07:26.643238 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.643207 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.643580 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.643254 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.643580 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.643349 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.643580 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.643457 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.643580 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.643513 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.643812 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.643631 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.643812 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.643675 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2713fe-2336-4f1e-8013-f8ede390211d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.643812 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.643722 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2fwz\" (UniqueName: \"kubernetes.io/projected/fd2713fe-2336-4f1e-8013-f8ede390211d-kube-api-access-d2fwz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.643812 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.643761 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745047 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745014 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745228 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745096 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745228 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745137 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745228 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745185 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745228 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745220 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745438 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745277 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745438 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745333 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745438 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745366 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2713fe-2336-4f1e-8013-f8ede390211d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745438 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745420 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fwz\" (UniqueName: \"kubernetes.io/projected/fd2713fe-2336-4f1e-8013-f8ede390211d-kube-api-access-d2fwz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745680 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745656 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745748 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745710 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745748 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745730 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.745841 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.745795 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.746155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.746136 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2713fe-2336-4f1e-8013-f8ede390211d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.747737 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.747714 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.748071 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.748052 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.770033 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.770008 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2fwz\" (UniqueName: \"kubernetes.io/projected/fd2713fe-2336-4f1e-8013-f8ede390211d-kube-api-access-d2fwz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.770139 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.770030 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2713fe-2336-4f1e-8013-f8ede390211d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-nkx8k\" (UID: \"fd2713fe-2336-4f1e-8013-f8ede390211d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:26.904408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:26.904306 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:27.224985 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:27.224961 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k"] Nov 03 06:07:27.227266 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:07:27.227240 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2713fe_2336_4f1e_8013_f8ede390211d.slice/crio-267498086f03921a0cee3ff31c85ae9dbe66af17d2969d45a7bd01a908701a58 WatchSource:0}: Error finding container 267498086f03921a0cee3ff31c85ae9dbe66af17d2969d45a7bd01a908701a58: Status 404 returned error can't find the container with id 267498086f03921a0cee3ff31c85ae9dbe66af17d2969d45a7bd01a908701a58 Nov 03 06:07:27.229632 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:27.229599 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:07:27.229713 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:27.229665 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:07:27.229713 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:27.229692 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:07:28.111554 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:28.108681 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" event={"ID":"fd2713fe-2336-4f1e-8013-f8ede390211d","Type":"ContainerStarted","Data":"6a578132c38680d157b52dfc76e18785c6cbdf498078f2803b26ff74e467a34f"} Nov 03 06:07:28.111554 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:28.108742 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" event={"ID":"fd2713fe-2336-4f1e-8013-f8ede390211d","Type":"ContainerStarted","Data":"267498086f03921a0cee3ff31c85ae9dbe66af17d2969d45a7bd01a908701a58"} Nov 03 06:07:28.199931 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:28.199879 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" podStartSLOduration=2.199865887 podStartE2EDuration="2.199865887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:28.186297212 +0000 UTC m=+735.994684480" watchObservedRunningTime="2026-11-03 06:07:28.199865887 +0000 UTC m=+736.008253154" Nov 03 06:07:28.545777 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:28.545749 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:07:28.884445 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:07:28.884342 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:07:28.904754 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:28.904723 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:28.906371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:28.906347 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nkx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 06:07:28.906503 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:28.906414 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" podUID="fd2713fe-2336-4f1e-8013-f8ede390211d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 06:07:29.905906 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:29.905783 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nkx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 06:07:29.906883 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:29.906847 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" podUID="fd2713fe-2336-4f1e-8013-f8ede390211d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 06:07:30.041141 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.041108 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd"] Nov 03 06:07:30.056253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.056227 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.079758 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.079718 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b7e447\"/\"kube-root-ca.crt\"" Nov 03 06:07:30.080086 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.080069 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b7e447\"/\"openshift-service-ca.crt\"" Nov 03 06:07:30.099246 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.099219 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b7e447\"/\"distinct-w3-b7e447-kserve-self-signed-certs\"" Nov 03 06:07:30.153569 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.107929 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd"] Nov 03 06:07:30.153865 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.111572 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b7e447\"/\"default-dockercfg-xxndc\"" Nov 03 06:07:30.159093 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.112187 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n25gk\" (UniqueName: \"kubernetes.io/projected/c2c59a2e-54d8-497a-a4ce-37039429fde0-kube-api-access-n25gk\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.168610 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.168560 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c59a2e-54d8-497a-a4ce-37039429fde0-tls-certs\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.169024 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.169003 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-home\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.189483 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.189443 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-tmp-dir\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.199197 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.199152 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-model-cache\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.212742 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.212702 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-dshm\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.389988 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.389947 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-home\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.390947 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.390892 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-tmp-dir\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.419401 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.390670 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-home\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.425607 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.399345 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-tmp-dir\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.429632 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.429605 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-model-cache\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.440143 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.419284 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-model-cache\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.455934 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.455883 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-dshm\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.456733 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.456706 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25gk\" (UniqueName: \"kubernetes.io/projected/c2c59a2e-54d8-497a-a4ce-37039429fde0-kube-api-access-n25gk\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.474785 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.474733 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c59a2e-54d8-497a-a4ce-37039429fde0-tls-certs\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.550801 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.549736 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-dshm\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.551477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.551442 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c59a2e-54d8-497a-a4ce-37039429fde0-tls-certs\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.552223 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.552202 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25gk\" (UniqueName: \"kubernetes.io/projected/c2c59a2e-54d8-497a-a4ce-37039429fde0-kube-api-access-n25gk\") pod \"distinct-w3-b7e447-kserve-c8c66c745-6vpdd\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.764080 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.762856 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:30.908315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.907867 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nkx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 06:07:30.908315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:30.907928 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" podUID="fd2713fe-2336-4f1e-8013-f8ede390211d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 06:07:31.569562 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:31.569523 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd"] Nov 03 06:07:31.570106 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:07:31.570077 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2c59a2e_54d8_497a_a4ce_37039429fde0.slice/crio-e893ab16225e80bd21ac3874f96030323689756cde513479cf3f8e0a231cb65d WatchSource:0}: Error finding container e893ab16225e80bd21ac3874f96030323689756cde513479cf3f8e0a231cb65d: Status 404 returned error can't find the container with id e893ab16225e80bd21ac3874f96030323689756cde513479cf3f8e0a231cb65d Nov 03 06:07:31.905406 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:31.905293 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nkx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 06:07:31.905406 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:31.905352 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" podUID="fd2713fe-2336-4f1e-8013-f8ede390211d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 06:07:32.388973 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:32.388933 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" event={"ID":"c2c59a2e-54d8-497a-a4ce-37039429fde0","Type":"ContainerStarted","Data":"40554fc8e4d31302e67c21eb923e9735bcfff2190b5d001d1cbb0faaedf3f773"} Nov 03 06:07:32.389468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:32.388982 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" event={"ID":"c2c59a2e-54d8-497a-a4ce-37039429fde0","Type":"ContainerStarted","Data":"e893ab16225e80bd21ac3874f96030323689756cde513479cf3f8e0a231cb65d"} Nov 03 06:07:32.458787 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:32.458715 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podStartSLOduration=2.458695831 podStartE2EDuration="2.458695831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:32.456848315 +0000 UTC m=+740.265235582" watchObservedRunningTime="2026-11-03 06:07:32.458695831 +0000 UTC m=+740.267083101" Nov 03 06:07:32.763647 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:32.763562 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:32.905383 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:32.905345 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nkx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 06:07:32.905597 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:32.905437 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" podUID="fd2713fe-2336-4f1e-8013-f8ede390211d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 06:07:33.904750 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:33.904716 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nkx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 06:07:33.905187 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:33.904766 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" podUID="fd2713fe-2336-4f1e-8013-f8ede390211d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 06:07:34.904849 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:34.904809 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nkx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 06:07:34.905267 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:34.904888 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" podUID="fd2713fe-2336-4f1e-8013-f8ede390211d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 06:07:36.210519 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:36.210483 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:36.211080 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:36.211052 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:36.211654 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:36.211635 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" Nov 03 06:07:38.905729 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.905697 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz"] Nov 03 06:07:38.910193 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.910169 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:38.949115 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.949079 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz"] Nov 03 06:07:38.973599 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.973567 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9sft\" (UniqueName: \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-kube-api-access-g9sft\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:38.973755 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.973626 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:38.973755 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.973714 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:38.973755 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.973734 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:38.973933 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.973912 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:38.973968 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.973946 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:38.973999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.973979 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:38.974043 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.974006 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:38.974043 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:38.974035 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075188 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075150 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9sft\" (UniqueName: \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-kube-api-access-g9sft\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075428 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075246 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075428 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075271 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075428 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075296 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075461 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075498 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075537 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075576 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075611 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075890 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075773 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.075890 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075818 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.076005 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.075909 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.076131 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.076103 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.076347 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.076331 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.077781 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.077761 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.077905 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.077886 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.099949 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.099922 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.100062 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.100044 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9sft\" (UniqueName: \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-kube-api-access-g9sft\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.225717 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.225627 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:39.601495 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.601437 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz"] Nov 03 06:07:39.604083 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:07:39.604055 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad38dc0_f80c_4c16_bd8f_ecb23a35c349.slice/crio-96359097855bc2987eeb77f5cf2bdcb3e51e1af4ff63c856eb89e985cabde811 WatchSource:0}: Error finding container 96359097855bc2987eeb77f5cf2bdcb3e51e1af4ff63c856eb89e985cabde811: Status 404 returned error can't find the container with id 96359097855bc2987eeb77f5cf2bdcb3e51e1af4ff63c856eb89e985cabde811 Nov 03 06:07:39.606558 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.606524 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:07:39.606664 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.606604 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:07:39.606664 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:39.606651 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:07:40.289997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.289963 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn"] Nov 03 06:07:40.294320 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.294292 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.325665 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.325637 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-df7236\"/\"kube-root-ca.crt\"" Nov 03 06:07:40.326020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.325995 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-df7236\"/\"openshift-service-ca.crt\"" Nov 03 06:07:40.326341 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.326211 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df7236\"/\"default-dockercfg-9pvnm\"" Nov 03 06:07:40.326443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.326286 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df7236\"/\"distinct-w0-df7236-kserve-self-signed-certs\"" Nov 03 06:07:40.357038 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.357004 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn"] Nov 03 06:07:40.389953 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.389920 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tmp-dir\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.390119 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.389983 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-home\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.390119 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.390050 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-model-cache\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.390197 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.390157 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tls-certs\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.390252 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.390209 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-dshm\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.390301 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.390262 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c5f4\" (UniqueName: \"kubernetes.io/projected/f80d1e19-3ce6-4060-8305-7cfec3f4c296-kube-api-access-2c5f4\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.461536 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.461496 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" event={"ID":"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349","Type":"ContainerStarted","Data":"dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310"} Nov 03 06:07:40.461694 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.461552 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" event={"ID":"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349","Type":"ContainerStarted","Data":"96359097855bc2987eeb77f5cf2bdcb3e51e1af4ff63c856eb89e985cabde811"} Nov 03 06:07:40.494340 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.494230 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tmp-dir\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.494634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.494357 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-home\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.494634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.494430 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-model-cache\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.494634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.494536 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tls-certs\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.494634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.494588 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-dshm\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.494634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.494613 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c5f4\" (UniqueName: \"kubernetes.io/projected/f80d1e19-3ce6-4060-8305-7cfec3f4c296-kube-api-access-2c5f4\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.497770 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.495859 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tmp-dir\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.497770 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.496442 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-home\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.497770 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.496700 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-model-cache\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.528504 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.528471 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tls-certs\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.531879 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.531847 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-dshm\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.546463 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.543245 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c5f4\" (UniqueName: \"kubernetes.io/projected/f80d1e19-3ce6-4060-8305-7cfec3f4c296-kube-api-access-2c5f4\") pod \"distinct-w0-df7236-kserve-786cb9dbc-98rdn\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.617442 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.616605 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:40.764632 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.764589 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:40.778365 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.778337 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:40.824057 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.823962 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" podStartSLOduration=2.82394729 podStartE2EDuration="2.82394729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:40.529158859 +0000 UTC m=+748.337546153" watchObservedRunningTime="2026-11-03 06:07:40.82394729 +0000 UTC m=+748.632334550" Nov 03 06:07:40.958806 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:40.958781 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn"] Nov 03 06:07:40.960558 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:07:40.960535 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf80d1e19_3ce6_4060_8305_7cfec3f4c296.slice/crio-400d3d57bed13e400312975f2bb918e28fcbac500ebf1ef6b36fc11f60744e19 WatchSource:0}: Error finding container 400d3d57bed13e400312975f2bb918e28fcbac500ebf1ef6b36fc11f60744e19: Status 404 returned error can't find the container with id 400d3d57bed13e400312975f2bb918e28fcbac500ebf1ef6b36fc11f60744e19 Nov 03 06:07:41.225854 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:41.225768 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:41.227370 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:41.227347 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 06:07:41.227504 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:41.227421 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 06:07:41.468838 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:41.468805 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" event={"ID":"f80d1e19-3ce6-4060-8305-7cfec3f4c296","Type":"ContainerStarted","Data":"98020769a3dbbce0f9c7eac3f06d68cc0aaffac8400e053c094707f2ff860dab"} Nov 03 06:07:41.468838 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:41.468845 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" event={"ID":"f80d1e19-3ce6-4060-8305-7cfec3f4c296","Type":"ContainerStarted","Data":"400d3d57bed13e400312975f2bb918e28fcbac500ebf1ef6b36fc11f60744e19"} Nov 03 06:07:41.480129 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:41.480059 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:07:41.512740 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:41.512689 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podStartSLOduration=1.512675483 podStartE2EDuration="1.512675483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:41.511584389 +0000 UTC m=+749.319971657" watchObservedRunningTime="2026-11-03 06:07:41.512675483 +0000 UTC m=+749.321062751" Nov 03 06:07:41.617110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:41.617076 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:42.226694 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:42.226662 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 06:07:42.226894 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:42.226712 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 06:07:43.225937 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:43.225900 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 06:07:43.226442 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:43.225959 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 06:07:44.226648 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:44.226610 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 06:07:44.227046 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:44.226660 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 06:07:45.226362 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.226327 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 06:07:45.226551 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.226381 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 06:07:45.402618 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.402581 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8"] Nov 03 06:07:45.407731 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.407707 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.424150 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.424125 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8"] Nov 03 06:07:45.455414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.452780 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-86728b\"/\"distinct-w6-86728b-kserve-self-signed-certs\"" Nov 03 06:07:45.455414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.452968 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-86728b\"/\"default-dockercfg-r27rx\"" Nov 03 06:07:45.462410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.462134 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-86728b\"/\"openshift-service-ca.crt\"" Nov 03 06:07:45.462410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.462253 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-86728b\"/\"kube-root-ca.crt\"" Nov 03 06:07:45.513836 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.513749 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f0b7ce-07b2-43be-819f-cde40d902902-tls-certs\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.513996 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.513843 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-tmp-dir\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.513996 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.513882 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-dshm\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.513996 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.513977 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-model-cache\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.514099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.514045 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7s9\" (UniqueName: \"kubernetes.io/projected/b7f0b7ce-07b2-43be-819f-cde40d902902-kube-api-access-ph7s9\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.514099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.514074 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-home\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.620170 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.620125 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-tmp-dir\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.620470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.620445 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-dshm\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.620702 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.620683 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-model-cache\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.620902 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.620883 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7s9\" (UniqueName: \"kubernetes.io/projected/b7f0b7ce-07b2-43be-819f-cde40d902902-kube-api-access-ph7s9\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.621058 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.621042 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-home\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.621193 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.621178 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f0b7ce-07b2-43be-819f-cde40d902902-tls-certs\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.636404 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.635818 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-model-cache\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.636404 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.636079 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-tmp-dir\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.637276 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.637247 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-home\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.645993 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.645952 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-dshm\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.646352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.646328 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f0b7ce-07b2-43be-819f-cde40d902902-tls-certs\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.669060 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.669023 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7s9\" (UniqueName: \"kubernetes.io/projected/b7f0b7ce-07b2-43be-819f-cde40d902902-kube-api-access-ph7s9\") pod \"distinct-w6-86728b-kserve-6894c57c46-pg7t8\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:45.728911 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:45.728872 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:46.043076 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:46.043042 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8"] Nov 03 06:07:46.436470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:46.436440 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:46.438419 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:46.437869 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:46.439082 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:46.439057 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:07:46.501159 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:46.501116 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" event={"ID":"b7f0b7ce-07b2-43be-819f-cde40d902902","Type":"ContainerStarted","Data":"eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465"} Nov 03 06:07:46.501342 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:46.501166 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" event={"ID":"b7f0b7ce-07b2-43be-819f-cde40d902902","Type":"ContainerStarted","Data":"3116c1277a9d7c693bbc1e8c97bf2af78ddb7af3b8cbb067593ddf6e266bc272"} Nov 03 06:07:46.601425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:46.601343 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podStartSLOduration=1.601324125 podStartE2EDuration="1.601324125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:46.598212778 +0000 UTC m=+754.406600050" watchObservedRunningTime="2026-11-03 06:07:46.601324125 +0000 UTC m=+754.409711395" Nov 03 06:07:46.729024 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:46.728945 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:50.620215 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:50.619002 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:50.843408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:50.843360 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:51.550283 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:51.550255 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:07:54.373568 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.373497 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm"] Nov 03 06:07:54.377769 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.377752 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.411630 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.411604 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm"] Nov 03 06:07:54.454128 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.454086 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.454331 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.454214 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.454331 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.454290 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.454534 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.454345 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrkh\" (UniqueName: \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-kube-api-access-xdrkh\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.454534 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.454437 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.454534 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.454474 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.454534 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.454532 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.455683 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.455657 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.456369 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.456342 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.557684 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.557646 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.557855 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.557804 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.557895 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.557855 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrkh\" (UniqueName: \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-kube-api-access-xdrkh\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.557941 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.557902 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.557941 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.557922 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.558046 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.557958 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.558046 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.558039 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.558152 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.558050 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.558152 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.558137 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.558258 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.558146 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.558258 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.558213 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.558344 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.558249 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.558476 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.558458 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.558678 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.558655 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.560294 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.560269 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.560859 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.560835 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.582099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.582072 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.582351 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.582325 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrkh\" (UniqueName: \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-kube-api-access-xdrkh\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:54.693017 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:54.692935 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:55.011932 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:55.011907 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm"] Nov 03 06:07:55.014475 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:07:55.014445 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fbec74_f9e1_4d46_abcc_cedad8e517fd.slice/crio-88484b362aa5b11c2299fa8ee76e72bd22c83ab7a61ed83dbbdc280ca1b305df WatchSource:0}: Error finding container 88484b362aa5b11c2299fa8ee76e72bd22c83ab7a61ed83dbbdc280ca1b305df: Status 404 returned error can't find the container with id 88484b362aa5b11c2299fa8ee76e72bd22c83ab7a61ed83dbbdc280ca1b305df Nov 03 06:07:55.016886 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:55.016849 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:07:55.016957 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:55.016923 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:07:55.017000 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:55.016960 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:07:55.605150 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:55.605108 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" event={"ID":"c4fbec74-f9e1-4d46-abcc-cedad8e517fd","Type":"ContainerStarted","Data":"ed53830283109f24d4dc6403b86c5ddb661aebd8d517b705099bd6410be4b49f"} Nov 03 06:07:55.605571 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:55.605172 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" event={"ID":"c4fbec74-f9e1-4d46-abcc-cedad8e517fd","Type":"ContainerStarted","Data":"88484b362aa5b11c2299fa8ee76e72bd22c83ab7a61ed83dbbdc280ca1b305df"} Nov 03 06:07:55.660455 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:55.660367 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" podStartSLOduration=1.660344963 podStartE2EDuration="1.660344963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:55.656678338 +0000 UTC m=+763.465065607" watchObservedRunningTime="2026-11-03 06:07:55.660344963 +0000 UTC m=+763.468732232" Nov 03 06:07:55.693897 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:55.693857 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:07:55.729269 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:55.729233 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:55.770176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:55.770143 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:56.622052 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:56.622018 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:07:56.695146 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:56.695104 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 06:07:56.695311 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:56.695156 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 06:07:57.693371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:57.693336 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 06:07:57.693884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:57.693416 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 06:07:58.693934 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:58.693897 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 06:07:58.694324 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:58.693970 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 06:07:58.884705 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:07:58.884659 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:07:59.694996 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:59.694951 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 06:07:59.703059 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:07:59.703015 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 06:08:00.693378 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:00.693346 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 06:08:00.693581 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:00.693409 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 06:08:02.045935 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:02.045896 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:08:02.047529 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:02.046622 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:08:02.048710 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:02.048689 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:08:06.699804 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:06.699766 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:08:09.691820 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.691788 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg"] Nov 03 06:08:09.697221 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.697198 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.717570 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.717545 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg"] Nov 03 06:08:09.765983 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.765939 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.765983 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.765980 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.766236 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.766075 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.766236 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.766130 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.766236 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.766153 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74571913-1a5a-4f8a-bf23-68211e1935d3-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.766236 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.766226 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.766445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.766272 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.766445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.766349 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.766445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.766414 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjf2\" (UniqueName: \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-kube-api-access-2vjf2\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.867491 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.867458 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjf2\" (UniqueName: \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-kube-api-access-2vjf2\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.867690 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.867546 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.867690 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.867575 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.867690 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.867625 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.867690 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.867660 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.867690 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.867683 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74571913-1a5a-4f8a-bf23-68211e1935d3-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.867963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.867737 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.867963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.867811 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.867963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.867902 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.868136 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.868110 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.868199 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.868155 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.868199 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.868175 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.868327 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.868307 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.868502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.868484 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74571913-1a5a-4f8a-bf23-68211e1935d3-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.870219 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.870196 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.870332 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.870313 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.891983 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.891954 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:09.892106 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:09.892043 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjf2\" (UniqueName: \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-kube-api-access-2vjf2\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:10.011529 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:10.011443 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:10.336167 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:10.336143 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg"] Nov 03 06:08:10.338017 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:08:10.337989 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74571913_1a5a_4f8a_bf23_68211e1935d3.slice/crio-b6780409e4a02b4dbc90e51736e0fdb0b1ecfce89697a74023e10d242871a87a WatchSource:0}: Error finding container b6780409e4a02b4dbc90e51736e0fdb0b1ecfce89697a74023e10d242871a87a: Status 404 returned error can't find the container with id b6780409e4a02b4dbc90e51736e0fdb0b1ecfce89697a74023e10d242871a87a Nov 03 06:08:10.340248 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:10.340217 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:10.340319 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:10.340282 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:10.340319 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:10.340310 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:10.737031 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:10.736942 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" event={"ID":"74571913-1a5a-4f8a-bf23-68211e1935d3","Type":"ContainerStarted","Data":"c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65"} Nov 03 06:08:10.737031 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:10.737006 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" event={"ID":"74571913-1a5a-4f8a-bf23-68211e1935d3","Type":"ContainerStarted","Data":"b6780409e4a02b4dbc90e51736e0fdb0b1ecfce89697a74023e10d242871a87a"} Nov 03 06:08:10.795427 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:10.789914 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" podStartSLOduration=1.789891476 podStartE2EDuration="1.789891476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:10.784177571 +0000 UTC m=+778.592564836" watchObservedRunningTime="2026-11-03 06:08:10.789891476 +0000 UTC m=+778.598278744" Nov 03 06:08:11.011770 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:11.011686 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:11.013401 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:11.013370 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 06:08:11.013519 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:11.013442 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 06:08:12.012437 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:12.011826 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 06:08:12.012437 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:12.011878 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 06:08:13.012470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:13.012435 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 06:08:13.012946 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:13.012488 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 06:08:14.012818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:14.012777 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 06:08:14.013245 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:14.012845 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 06:08:15.011956 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:15.011924 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 06:08:15.012110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:15.011976 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 06:08:16.011857 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:16.011820 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 06:08:16.012253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:16.011873 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 06:08:17.245550 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:17.245514 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:17.246621 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:17.246596 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:17.247374 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:17.247355 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:08:20.203053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.203023 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l"] Nov 03 06:08:20.208983 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.208958 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.227510 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.227483 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l"] Nov 03 06:08:20.236657 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.236631 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b7e447\"/\"distinct-2-w3-b7e447-kserve-self-signed-certs\"" Nov 03 06:08:20.345085 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.345052 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tls-certs\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.345244 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.345186 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tmp-dir\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.345383 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.345363 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-model-cache\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.345452 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.345420 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-home\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.345491 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.345453 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-dshm\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.345525 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.345490 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62xpq\" (UniqueName: \"kubernetes.io/projected/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-kube-api-access-62xpq\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.447209 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.447164 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62xpq\" (UniqueName: \"kubernetes.io/projected/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-kube-api-access-62xpq\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.448044 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.448017 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tls-certs\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.448891 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.448869 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tmp-dir\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.449666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.449644 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-model-cache\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.466917 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.466838 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-home\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.467535 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.467507 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-dshm\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.467813 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.455596 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tls-certs\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.467942 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.449381 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tmp-dir\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.468053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.467450 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-home\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.468152 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.450863 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-model-cache\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.479797 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.479767 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-dshm\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.484157 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.484128 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62xpq\" (UniqueName: \"kubernetes.io/projected/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-kube-api-access-62xpq\") pod \"distinct-2-w3-b7e447-kserve-54cf4469db-dp92l\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.529648 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.529617 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:20.871867 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:20.871832 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l"] Nov 03 06:08:20.876031 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:08:20.875999 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14b4bbcb_54da_4bd5_a7ed_7e1e276a58bc.slice/crio-d78158cfd402faebcdd34bf653fb4e02431dd6c735c1ead0d7138f4e210c71ac WatchSource:0}: Error finding container d78158cfd402faebcdd34bf653fb4e02431dd6c735c1ead0d7138f4e210c71ac: Status 404 returned error can't find the container with id d78158cfd402faebcdd34bf653fb4e02431dd6c735c1ead0d7138f4e210c71ac Nov 03 06:08:21.924500 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:21.924087 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" event={"ID":"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc","Type":"ContainerStarted","Data":"a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc"} Nov 03 06:08:21.924500 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:21.924130 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" event={"ID":"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc","Type":"ContainerStarted","Data":"d78158cfd402faebcdd34bf653fb4e02431dd6c735c1ead0d7138f4e210c71ac"} Nov 03 06:08:22.532090 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.531602 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:22.715818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.715756 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podStartSLOduration=2.715736819 podStartE2EDuration="2.715736819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:22.013929291 +0000 UTC m=+789.822316562" watchObservedRunningTime="2026-11-03 06:08:22.715736819 +0000 UTC m=+790.524124088" Nov 03 06:08:22.718543 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.718515 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj"] Nov 03 06:08:22.723384 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.723363 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.742861 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.742833 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj"] Nov 03 06:08:22.748127 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.748099 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-home\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.748275 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.748257 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg8z6\" (UniqueName: \"kubernetes.io/projected/a6892c41-6866-43ee-82da-594bc4e28b29-kube-api-access-dg8z6\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.748340 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.748321 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6892c41-6866-43ee-82da-594bc4e28b29-tls-certs\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.748503 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.748481 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-model-cache\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.748554 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.748534 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-tmp-dir\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.748663 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.748647 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-dshm\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.751016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.750993 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9688d8\"/\"distinct-w1-9688d8-kserve-self-signed-certs\"" Nov 03 06:08:22.751125 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.751060 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9688d8\"/\"kube-root-ca.crt\"" Nov 03 06:08:22.753076 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.753053 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9688d8\"/\"openshift-service-ca.crt\"" Nov 03 06:08:22.753137 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.753122 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9688d8\"/\"default-dockercfg-zgwhm\"" Nov 03 06:08:22.850364 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.850330 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-model-cache\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.850606 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.850374 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-tmp-dir\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.850606 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.850479 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-dshm\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.850606 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.850545 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-home\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.850778 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.850655 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg8z6\" (UniqueName: \"kubernetes.io/projected/a6892c41-6866-43ee-82da-594bc4e28b29-kube-api-access-dg8z6\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.850778 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.850715 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6892c41-6866-43ee-82da-594bc4e28b29-tls-certs\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.850878 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.850846 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-model-cache\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.850934 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.850879 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-tmp-dir\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.851077 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.851056 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-home\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.853076 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.853049 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-dshm\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.853197 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.853154 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6892c41-6866-43ee-82da-594bc4e28b29-tls-certs\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:22.875277 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:22.875255 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg8z6\" (UniqueName: \"kubernetes.io/projected/a6892c41-6866-43ee-82da-594bc4e28b29-kube-api-access-dg8z6\") pod \"distinct-w1-9688d8-kserve-7599d77959-jrjvj\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:23.044107 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:23.044035 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:23.353674 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:23.353648 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj"] Nov 03 06:08:23.354818 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:08:23.354797 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6892c41_6866_43ee_82da_594bc4e28b29.slice/crio-7c88f1ca840b354763b7bf7ba65ff624c88afd0937aeddbfb2044ef959390038 WatchSource:0}: Error finding container 7c88f1ca840b354763b7bf7ba65ff624c88afd0937aeddbfb2044ef959390038: Status 404 returned error can't find the container with id 7c88f1ca840b354763b7bf7ba65ff624c88afd0937aeddbfb2044ef959390038 Nov 03 06:08:23.940923 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:23.940886 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" event={"ID":"a6892c41-6866-43ee-82da-594bc4e28b29","Type":"ContainerStarted","Data":"0240ac82bbf82e1998b19860048bc821dabb1eaf7d45e60c017e64ca68c3ccb2"} Nov 03 06:08:23.940923 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:23.940929 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" event={"ID":"a6892c41-6866-43ee-82da-594bc4e28b29","Type":"ContainerStarted","Data":"7c88f1ca840b354763b7bf7ba65ff624c88afd0937aeddbfb2044ef959390038"} Nov 03 06:08:23.984718 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:23.984668 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podStartSLOduration=1.984654832 podStartE2EDuration="1.984654832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:23.983602972 +0000 UTC m=+791.791990239" watchObservedRunningTime="2026-11-03 06:08:23.984654832 +0000 UTC m=+791.793042099" Nov 03 06:08:24.044547 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.044507 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:24.056803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.056770 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh"] Nov 03 06:08:24.060725 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.060708 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.083483 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.083452 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh"] Nov 03 06:08:24.179995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.178196 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.179995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.178293 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.179995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.178335 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.179995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.178364 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.179995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.178503 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea987416-f4be-475c-8574-903796fcf95d-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.179995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.178548 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea987416-f4be-475c-8574-903796fcf95d-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.179995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.178609 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.179995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.178632 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbbcb\" (UniqueName: \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-kube-api-access-zbbcb\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.179995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.178710 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.279799 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.279711 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea987416-f4be-475c-8574-903796fcf95d-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.279799 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.279762 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea987416-f4be-475c-8574-903796fcf95d-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280014 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.279806 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280014 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.279824 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbbcb\" (UniqueName: \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-kube-api-access-zbbcb\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280014 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.279895 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280161 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.280054 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280161 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.280126 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.280165 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.280192 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280520 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.280493 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280627 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.280586 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280627 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.280600 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280766 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.280726 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea987416-f4be-475c-8574-903796fcf95d-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.280766 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.280744 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.282405 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.282368 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.282579 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.282560 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea987416-f4be-475c-8574-903796fcf95d-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.304283 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.304256 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.304425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.304259 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbbcb\" (UniqueName: \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-kube-api-access-zbbcb\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.383631 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.383601 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:24.709976 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.708189 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh"] Nov 03 06:08:24.764306 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.762705 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:24.764306 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.762792 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:24.764306 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.762837 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:24.949185 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.949147 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" event={"ID":"ea987416-f4be-475c-8574-903796fcf95d","Type":"ContainerStarted","Data":"24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb"} Nov 03 06:08:24.949361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.949212 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" event={"ID":"ea987416-f4be-475c-8574-903796fcf95d","Type":"ContainerStarted","Data":"5e79946052c04d120ee141443c3fdaccf12a12440b35987d0edca9ceaebcbbfc"} Nov 03 06:08:24.998600 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:24.998509 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podStartSLOduration=0.998496828 podStartE2EDuration="998.496828ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:24.996140091 +0000 UTC m=+792.804527359" watchObservedRunningTime="2026-11-03 06:08:24.998496828 +0000 UTC m=+792.806884097" Nov 03 06:08:25.384359 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:25.384320 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:25.386051 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:25.386026 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 06:08:25.386161 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:25.386071 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 06:08:26.384277 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:26.384237 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 06:08:26.384489 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:26.384296 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 06:08:27.384360 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:27.384324 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 06:08:27.384584 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:27.384378 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 06:08:28.384836 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:28.384798 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 06:08:28.385451 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:28.385415 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 06:08:28.883996 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:08:28.883963 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:08:29.384729 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:29.384695 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 06:08:29.384912 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:29.384749 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 06:08:30.384491 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:30.384455 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 06:08:30.384730 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:30.384511 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 06:08:30.530322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:30.530288 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:30.569196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:30.569165 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:30.999893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:30.999866 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:31.384339 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:31.384299 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 06:08:31.384546 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:31.384371 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 06:08:32.388139 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:32.388110 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:32.388577 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:32.388432 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:32.388997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:32.388978 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:08:33.044568 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:33.044536 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:33.057138 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:33.057116 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:34.019347 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:34.019319 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:38.783772 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.783736 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll"] Nov 03 06:08:38.790758 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.790731 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:38.816942 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.807190 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll"] Nov 03 06:08:38.909802 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.909763 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:38.909979 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.909924 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66zbt\" (UniqueName: \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-kube-api-access-66zbt\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:38.909979 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.909973 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:38.910053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.909994 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a908b4b-a058-4eea-9d4b-05899552a98c-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:38.910096 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.910075 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:38.910131 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.910116 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:38.910234 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.910220 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:38.910274 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.910249 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:38.910308 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:38.910292 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.011457 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.011423 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66zbt\" (UniqueName: \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-kube-api-access-66zbt\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.011661 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.011495 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.011661 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.011536 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a908b4b-a058-4eea-9d4b-05899552a98c-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.011661 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.011567 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.011661 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.011598 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.011889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.011667 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.011889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.011705 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.011889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.011746 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.011889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.011802 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.012315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.012106 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.012315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.012186 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.012315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.012269 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.012560 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.012416 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.012560 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.012467 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a908b4b-a058-4eea-9d4b-05899552a98c-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.013934 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.013913 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.015248 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.015230 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.037537 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.037468 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66zbt\" (UniqueName: \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-kube-api-access-66zbt\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.040688 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.040665 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.133601 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.133570 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:39.435316 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.435282 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj"] Nov 03 06:08:39.441221 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.441201 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.454422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.454383 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj"] Nov 03 06:08:39.457168 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.457146 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be1a66\"/\"distinct-2-w4-be1a66-kserve-self-signed-certs\"" Nov 03 06:08:39.476912 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.476890 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll"] Nov 03 06:08:39.478509 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:08:39.478482 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a908b4b_a058_4eea_9d4b_05899552a98c.slice/crio-680aefbf56c73e1e7b900d58adfcfe9974b6f26b6afced0143d3ece43f24f2ee WatchSource:0}: Error finding container 680aefbf56c73e1e7b900d58adfcfe9974b6f26b6afced0143d3ece43f24f2ee: Status 404 returned error can't find the container with id 680aefbf56c73e1e7b900d58adfcfe9974b6f26b6afced0143d3ece43f24f2ee Nov 03 06:08:39.481247 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.481215 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:39.481349 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.481274 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:39.481349 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.481304 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:39.516744 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.516718 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-home\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.516863 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.516842 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tls-certs\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.516927 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.516906 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tmp-dir\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.516965 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.516950 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-dshm\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.516999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.516982 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-model-cache\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.517134 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.517115 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9qzb\" (UniqueName: \"kubernetes.io/projected/5e84841f-e79a-4bfd-a079-cd196a8c73e9-kube-api-access-m9qzb\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.529522 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.529499 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z"] Nov 03 06:08:39.534946 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.534927 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.559809 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.559778 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z"] Nov 03 06:08:39.626362 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.626253 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.626894 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.626866 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tls-certs\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.636321 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.636291 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tmp-dir\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.637628 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.637603 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-dshm\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.637870 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.637853 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-model-cache\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.638037 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.638023 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.638145 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.638133 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.638232 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.638220 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.638331 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.638319 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9qzb\" (UniqueName: \"kubernetes.io/projected/5e84841f-e79a-4bfd-a079-cd196a8c73e9-kube-api-access-m9qzb\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.638470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.638456 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc965\" (UniqueName: \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-kube-api-access-vc965\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.638769 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.638745 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.638979 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.638961 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-home\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.639113 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.639098 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.639253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.639238 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.639363 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.639350 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e2e8f6c-d325-4e8c-9f49-93477681daab-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.639966 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.636940 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tmp-dir\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.640699 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.640677 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-home\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.641134 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.641115 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-model-cache\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.647053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.646749 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tls-certs\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.695152 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.690265 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9qzb\" (UniqueName: \"kubernetes.io/projected/5e84841f-e79a-4bfd-a079-cd196a8c73e9-kube-api-access-m9qzb\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.702743 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.702709 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-dshm\") pod \"distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.746576 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.746443 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.747453 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.746832 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.751497 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.747633 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.753073 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.752690 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc965\" (UniqueName: \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-kube-api-access-vc965\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.753355 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.753235 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.753538 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.753511 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.753624 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.753585 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.753688 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.753622 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e2e8f6c-d325-4e8c-9f49-93477681daab-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.753688 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.753668 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.754374 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.754108 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.754374 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.754148 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.754638 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.747423 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.754845 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.754800 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e2e8f6c-d325-4e8c-9f49-93477681daab-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.755090 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.755069 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.755774 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.755614 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:39.756034 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.756007 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.756939 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.756919 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.782367 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.781900 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.783227 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.783182 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc965\" (UniqueName: \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-kube-api-access-vc965\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:39.858847 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:39.858811 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:40.079907 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:08:40.079879 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e84841f_e79a_4bfd_a079_cd196a8c73e9.slice/crio-a4c321f6c8f35cbd76bd025c247960552597028b7825e5e6ce36f076df093bbf WatchSource:0}: Error finding container a4c321f6c8f35cbd76bd025c247960552597028b7825e5e6ce36f076df093bbf: Status 404 returned error can't find the container with id a4c321f6c8f35cbd76bd025c247960552597028b7825e5e6ce36f076df093bbf Nov 03 06:08:40.080070 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.080029 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj"] Nov 03 06:08:40.082253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.082224 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" event={"ID":"6a908b4b-a058-4eea-9d4b-05899552a98c","Type":"ContainerStarted","Data":"16dd80b96b3f0e0b29eedb81d1b6667865540f07bc45c8a0d92016583bd07747"} Nov 03 06:08:40.082362 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.082277 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" event={"ID":"6a908b4b-a058-4eea-9d4b-05899552a98c","Type":"ContainerStarted","Data":"680aefbf56c73e1e7b900d58adfcfe9974b6f26b6afced0143d3ece43f24f2ee"} Nov 03 06:08:40.130517 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.130465 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podStartSLOduration=2.130452165 podStartE2EDuration="2.130452165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:40.128746279 +0000 UTC m=+807.937133590" watchObservedRunningTime="2026-11-03 06:08:40.130452165 +0000 UTC m=+807.938839432" Nov 03 06:08:40.133829 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.133804 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:40.138127 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.138092 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 06:08:40.138252 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.138145 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 06:08:40.177836 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.177807 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z"] Nov 03 06:08:40.180177 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:08:40.180142 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2e8f6c_d325_4e8c_9f49_93477681daab.slice/crio-2ac1ffc4de6cd395eec77d64a74fbebb1f5b46229d3bebebcad2810399909abf WatchSource:0}: Error finding container 2ac1ffc4de6cd395eec77d64a74fbebb1f5b46229d3bebebcad2810399909abf: Status 404 returned error can't find the container with id 2ac1ffc4de6cd395eec77d64a74fbebb1f5b46229d3bebebcad2810399909abf Nov 03 06:08:40.185467 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.183684 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:40.185467 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.183769 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:40.185467 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:40.183814 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:41.127833 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.127789 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" event={"ID":"5e84841f-e79a-4bfd-a079-cd196a8c73e9","Type":"ContainerStarted","Data":"27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f"} Nov 03 06:08:41.129459 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.129428 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" event={"ID":"5e84841f-e79a-4bfd-a079-cd196a8c73e9","Type":"ContainerStarted","Data":"a4c321f6c8f35cbd76bd025c247960552597028b7825e5e6ce36f076df093bbf"} Nov 03 06:08:41.133374 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.133343 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" event={"ID":"0e2e8f6c-d325-4e8c-9f49-93477681daab","Type":"ContainerStarted","Data":"1f1c0a835df895e23dedc8a235501fedda744f1ce2a51b0ece23e1a0005d5271"} Nov 03 06:08:41.134093 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.134071 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" event={"ID":"0e2e8f6c-d325-4e8c-9f49-93477681daab","Type":"ContainerStarted","Data":"2ac1ffc4de6cd395eec77d64a74fbebb1f5b46229d3bebebcad2810399909abf"} Nov 03 06:08:41.134248 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.134014 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 06:08:41.134430 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.134381 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 06:08:41.195412 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.195326 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podStartSLOduration=2.195300492 podStartE2EDuration="2.195300492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:41.191863613 +0000 UTC m=+809.000250882" watchObservedRunningTime="2026-11-03 06:08:41.195300492 +0000 UTC m=+809.003687762" Nov 03 06:08:41.763414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.759479 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:41.865329 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.865297 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:41.865704 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.865680 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 06:08:41.865876 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:41.865854 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 06:08:42.151429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:42.151301 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 06:08:42.151429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:42.151355 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 06:08:42.865957 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:42.865918 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 06:08:42.866232 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:42.866197 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 06:08:43.144546 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:43.144458 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 06:08:43.144785 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:43.144758 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 06:08:43.355688 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:43.355625 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podStartSLOduration=4.355605057 podStartE2EDuration="4.355605057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:41.256263216 +0000 UTC m=+809.064650486" watchObservedRunningTime="2026-11-03 06:08:43.355605057 +0000 UTC m=+811.163992327" Nov 03 06:08:43.382550 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:43.382485 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:08:43.860414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:43.859843 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 06:08:43.860414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:43.859909 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 06:08:44.134924 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:44.134829 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 06:08:44.135196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:44.135166 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 06:08:44.859853 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:44.859813 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 06:08:44.860485 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:44.860436 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 06:08:45.140962 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:45.140870 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 06:08:45.141215 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:45.141186 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 06:08:45.872413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:45.870453 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 06:08:45.872413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:45.870514 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 06:08:45.992366 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:45.990868 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 06:08:46.042562 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.042508 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:46.136893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.136795 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 06:08:46.137170 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.137138 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 06:08:46.196956 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.196869 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:08:46.272411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.269556 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:46.397550 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.396459 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:46.475556 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.475495 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:46.551413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.551357 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:46.627549 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.627489 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:46.763467 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.748521 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:46.886413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.872683 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 06:08:46.886413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.872769 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 06:08:46.991915 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.991566 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:47.002410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:46.992207 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:47.014508 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.009708 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:47.053131 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.053091 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:47.053961 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.053938 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:08:47.355066 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.301484 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:47.355066 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.302255 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:08:47.355066 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.305454 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:47.355066 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.306699 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:08:47.490861 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.433236 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:47.523668 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.494522 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:47.523668 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.521734 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:08:47.523668 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.521802 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:08:47.523668 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.523555 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:08:47.523668 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.523601 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:47.561376 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.561331 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:47.562043 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.562019 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:08:47.574904 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.574876 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 06:08:47.596501 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.596458 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 06:08:47.632593 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.632494 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:08:47.655199 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.637492 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:08:47.827235 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.826178 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:08:47.950069 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.945562 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 06:08:47.950069 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.945601 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:08:47.950069 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.945621 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 06:08:47.990060 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.989851 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:08:47.990060 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:47.989921 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:48.011474 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.011282 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:48.013139 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.012753 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:48.014429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.014400 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:08:48.054247 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.053454 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:08:48.170499 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.170465 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:08:48.170923 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.170901 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:08:48.171113 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.171098 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:08:48.186419 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.182864 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 06:08:48.186419 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.182927 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 06:08:48.337211 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.335786 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:08:48.349409 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.342186 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:48.369598 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.369541 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:08:48.369843 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.369618 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:08:48.609190 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.609017 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:08:48.609464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.609446 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:08:48.885901 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.864491 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 06:08:48.885901 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:48.879513 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 06:08:49.757050 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:49.757017 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:49.772149 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:49.772119 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:49.776818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:49.776793 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:49.777441 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:49.777417 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:49.779346 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:49.779330 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:08:49.886034 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:49.885999 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:49.886560 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:49.886500 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:49.888510 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:49.887961 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:08:50.590416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:50.589680 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:08:53.816052 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.816018 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb"] Nov 03 06:08:53.821601 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.821578 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:53.842505 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.842478 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb"] Nov 03 06:08:53.973857 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.973821 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:53.974046 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.973880 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:53.974046 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.974023 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:53.974046 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.974040 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:53.974215 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.974075 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:53.974215 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.974095 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:53.974215 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.974187 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:53.974334 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.974294 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7drqd\" (UniqueName: \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-kube-api-access-7drqd\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:53.974334 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:53.974334 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.076281 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076188 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.076281 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076251 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.076548 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076366 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.076548 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076420 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.076548 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076473 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.076548 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076500 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.076548 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076528 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.077102 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076582 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7drqd\" (UniqueName: \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-kube-api-access-7drqd\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.077102 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076644 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.077102 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076683 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.077102 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076688 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.077102 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.076975 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.077102 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.077053 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.077308 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.077123 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.078903 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.078878 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.079083 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.079064 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.101246 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.101209 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.101377 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.101342 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7drqd\" (UniqueName: \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-kube-api-access-7drqd\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.141311 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.141285 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:54.466062 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.466033 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb"] Nov 03 06:08:54.467210 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:08:54.467184 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cee8e8_09ff_49e8_89ee_04c82a0f50a9.slice/crio-4952e16084fb6f4adf86caef08b99ec385310180f3323d9d53028423a22f8d4a WatchSource:0}: Error finding container 4952e16084fb6f4adf86caef08b99ec385310180f3323d9d53028423a22f8d4a: Status 404 returned error can't find the container with id 4952e16084fb6f4adf86caef08b99ec385310180f3323d9d53028423a22f8d4a Nov 03 06:08:54.469798 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.469767 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:54.469887 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.469854 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:54.469887 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.469882 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:08:54.608183 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.608148 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" event={"ID":"87cee8e8-09ff-49e8-89ee-04c82a0f50a9","Type":"ContainerStarted","Data":"7794b81a70c72b7b909cf1adeb6035a3765690ea4c1c4c9287e1d02b472955cc"} Nov 03 06:08:54.608467 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.608216 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" event={"ID":"87cee8e8-09ff-49e8-89ee-04c82a0f50a9","Type":"ContainerStarted","Data":"4952e16084fb6f4adf86caef08b99ec385310180f3323d9d53028423a22f8d4a"} Nov 03 06:08:54.688594 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:54.688449 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" podStartSLOduration=1.68842903 podStartE2EDuration="1.68842903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:54.676835373 +0000 UTC m=+822.485222644" watchObservedRunningTime="2026-11-03 06:08:54.68842903 +0000 UTC m=+822.496816301" Nov 03 06:08:55.155083 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:55.155010 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:08:55.156742 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:55.156714 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 06:08:55.156871 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:55.156766 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 06:08:56.152440 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:56.150495 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 06:08:56.152440 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:56.150559 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 06:08:57.149415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:57.144768 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 06:08:57.149415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:57.144832 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 06:08:58.148415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:58.146752 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 06:08:58.148415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:58.146812 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 06:08:58.884327 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:08:58.884292 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:08:59.145470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:59.144466 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 06:08:59.145470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:08:59.144525 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 06:09:00.141857 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:00.141764 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 06:09:00.142548 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:00.142511 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 06:09:01.142055 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:01.142019 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 06:09:01.142532 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:01.142085 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 06:09:02.146876 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:02.146846 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:09:02.147315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:02.147265 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:09:02.147958 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:02.147940 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:09:08.162534 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:08.162480 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:08.163285 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:08.163253 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:08.163749 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:08.163726 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 06:09:08.251895 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:08.251849 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:09:08.398549 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:08.398501 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:08.535613 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:08.535498 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:08.582550 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:08.582479 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:08.829912 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:08.829870 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:08.889592 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:08.889539 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:09.007035 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.006996 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2"] Nov 03 06:09:09.029003 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.028955 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:09:09.064231 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.064201 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn"] Nov 03 06:09:09.080522 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.064550 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.103974 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.103945 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.104298 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.103831 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2"] Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.163477 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.163529 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.274403 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.274457 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.164588 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.191612 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.222692 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275114 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275188 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275220 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275248 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82859956-f5dd-43be-894b-2e78907f7505-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275278 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275318 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275368 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275440 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275476 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275554 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275631 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275673 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfsjn\" (UniqueName: \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-kube-api-access-cfsjn\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275726 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7pb\" (UniqueName: \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-kube-api-access-8r7pb\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275762 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275798 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275838 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82859956-f5dd-43be-894b-2e78907f7505-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.275952 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.260577 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.276344 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:09:09.281445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.276416 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:09:09.513513 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.493924 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:09.513513 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.494359 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.520970 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521031 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521068 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82859956-f5dd-43be-894b-2e78907f7505-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521101 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521149 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521202 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521251 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521294 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521358 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521452 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521504 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsjn\" (UniqueName: \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-kube-api-access-cfsjn\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521563 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7pb\" (UniqueName: \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-kube-api-access-8r7pb\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521597 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521635 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521678 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82859956-f5dd-43be-894b-2e78907f7505-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521782 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521819 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.521918 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.530634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.522493 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.556267 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.556207 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:09.556999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.556973 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:09:09.614203 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.605165 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.614203 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.606605 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.616403 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.615238 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82859956-f5dd-43be-894b-2e78907f7505-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.616403 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.615578 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.619662 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.619097 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:09:09.619662 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.619181 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:09:09.630786 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.620808 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.657411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.655612 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.657411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.655930 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.732007 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.731443 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.732007 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.731542 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:09:09.738031 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.737978 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:09:09.739352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.739237 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn"] Nov 03 06:09:09.801726 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.795710 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:09.840416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.815167 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7pb\" (UniqueName: \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-kube-api-access-8r7pb\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:09.906632 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.880502 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:09.906632 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.880820 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:09:09.906897 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.906728 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:09:09.931415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:09.907072 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:09:10.003410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.000275 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfsjn\" (UniqueName: \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-kube-api-access-cfsjn\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:10.091469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.085234 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:10.109922 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.097083 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:09:10.109922 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.097514 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:09:10.266219 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.208161 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:10.266219 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.241793 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:10.266219 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.247630 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82859956-f5dd-43be-894b-2e78907f7505-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:10.306275 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.306170 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:10.307887 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.307823 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:10.443565 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.443463 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd"] Nov 03 06:09:10.592725 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.592678 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:10.722478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.721950 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:10.722478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.722171 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:09:10.722478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.722336 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:10.734982 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.734949 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:10.753986 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.736543 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:10.777573 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.777534 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:10.929524 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.928299 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:10.948082 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.928205 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd"] Nov 03 06:09:10.948082 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.947835 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:09:10.948082 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.947885 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:09:10.948082 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.947931 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:09:10.948082 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.947962 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:09:10.952889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.952307 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:09:10.972221 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.972161 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:09:10.974740 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.974081 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:09:10.982914 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.980247 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl"] Nov 03 06:09:11.003318 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:10.999093 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:09:11.003318 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.000096 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.011674 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.011591 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl"] Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.044726 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.044782 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.044840 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1bc896-f654-4773-9226-b5b099bb604b-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.044896 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.044955 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.044991 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045062 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045144 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045178 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fdb858d-e18c-4d4b-abce-054fe2219165-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045235 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045289 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpwc6\" (UniqueName: \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-kube-api-access-tpwc6\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045332 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045427 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045522 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9745r\" (UniqueName: \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-kube-api-access-9745r\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045664 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045699 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1bc896-f654-4773-9226-b5b099bb604b-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045787 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.052471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.045877 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.147627 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.147591 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.147627 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.147632 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fdb858d-e18c-4d4b-abce-054fe2219165-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.147856 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.147749 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.147856 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.147812 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwc6\" (UniqueName: \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-kube-api-access-tpwc6\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.147984 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.147858 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.147984 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.147904 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.147984 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.147957 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9745r\" (UniqueName: \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-kube-api-access-9745r\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.148155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148035 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.148155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148067 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1bc896-f654-4773-9226-b5b099bb604b-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.148155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148107 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.148155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148125 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.148477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148161 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.148477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148225 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.148477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148270 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.148477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148338 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.148477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148465 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1bc896-f654-4773-9226-b5b099bb604b-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.148835 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148512 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fdb858d-e18c-4d4b-abce-054fe2219165-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.148835 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148563 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.148835 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148620 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.148835 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148638 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.148835 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148664 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.148835 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148664 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.148835 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148669 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.148835 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148768 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.149493 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.148878 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.149493 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.149134 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.149493 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.149170 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.149493 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.149483 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1bc896-f654-4773-9226-b5b099bb604b-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.153269 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.153243 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1bc896-f654-4773-9226-b5b099bb604b-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.154198 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.154116 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.155583 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.155558 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.156640 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.156612 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.172468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.172439 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwc6\" (UniqueName: \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-kube-api-access-tpwc6\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.172925 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.172904 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9745r\" (UniqueName: \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-kube-api-access-9745r\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.177179 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.177140 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.177179 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.177163 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.273867 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.273793 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:11.318891 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.318850 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2"] Nov 03 06:09:11.319853 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:09:11.319812 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b19d23e_3efa_4c72_a703_6610bd96a3c4.slice/crio-4165a5497f1321cb365c1dd85d348761ddab106b03490322b4eaced50809a526 WatchSource:0}: Error finding container 4165a5497f1321cb365c1dd85d348761ddab106b03490322b4eaced50809a526: Status 404 returned error can't find the container with id 4165a5497f1321cb365c1dd85d348761ddab106b03490322b4eaced50809a526 Nov 03 06:09:11.323255 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.323224 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.323366 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.323300 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.323366 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.323350 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.341892 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.341739 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn"] Nov 03 06:09:11.355020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.354992 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:11.355492 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:09:11.355456 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82859956_f5dd_43be_894b_2e78907f7505.slice/crio-7462b8c307b88cb0e91062e72ade10b7d8fcb81254a12d475cfabe6a0ee8f921 WatchSource:0}: Error finding container 7462b8c307b88cb0e91062e72ade10b7d8fcb81254a12d475cfabe6a0ee8f921: Status 404 returned error can't find the container with id 7462b8c307b88cb0e91062e72ade10b7d8fcb81254a12d475cfabe6a0ee8f921 Nov 03 06:09:11.358964 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.358931 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.359111 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.359017 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.359111 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.359067 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.657099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.657067 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd"] Nov 03 06:09:11.661939 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:09:11.661902 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdb858d_e18c_4d4b_abce_054fe2219165.slice/crio-a3371c995dc65dbe0a7eb7cbb70c6a0b12702392d95bcd55fbca7bb644865478 WatchSource:0}: Error finding container a3371c995dc65dbe0a7eb7cbb70c6a0b12702392d95bcd55fbca7bb644865478: Status 404 returned error can't find the container with id a3371c995dc65dbe0a7eb7cbb70c6a0b12702392d95bcd55fbca7bb644865478 Nov 03 06:09:11.665181 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.665138 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.665453 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.665430 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.665604 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.665583 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.925109 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.925011 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl"] Nov 03 06:09:11.933634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.933598 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.933748 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.933672 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.933748 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.933713 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:11.970555 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.970518 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" event={"ID":"ff1bc896-f654-4773-9226-b5b099bb604b","Type":"ContainerStarted","Data":"1807269992c355a72ab411451762eec99e6332052deaaa4f75a3ab167970396d"} Nov 03 06:09:11.977374 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.975363 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" event={"ID":"6b19d23e-3efa-4c72-a703-6610bd96a3c4","Type":"ContainerStarted","Data":"eb13878edd20bc24955729046363a55f7c55d6fd565180eba2f3df9678a623ab"} Nov 03 06:09:11.977374 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.975441 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" event={"ID":"6b19d23e-3efa-4c72-a703-6610bd96a3c4","Type":"ContainerStarted","Data":"4165a5497f1321cb365c1dd85d348761ddab106b03490322b4eaced50809a526"} Nov 03 06:09:11.983567 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.983528 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" event={"ID":"9fdb858d-e18c-4d4b-abce-054fe2219165","Type":"ContainerStarted","Data":"b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a"} Nov 03 06:09:11.983747 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.983728 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" event={"ID":"9fdb858d-e18c-4d4b-abce-054fe2219165","Type":"ContainerStarted","Data":"a3371c995dc65dbe0a7eb7cbb70c6a0b12702392d95bcd55fbca7bb644865478"} Nov 03 06:09:11.986211 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.986187 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" event={"ID":"82859956-f5dd-43be-894b-2e78907f7505","Type":"ContainerStarted","Data":"c099a8b6c465176dc88af56014616d5c8cd601520b6d2832db31e643e19861ad"} Nov 03 06:09:11.986358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:11.986345 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" event={"ID":"82859956-f5dd-43be-894b-2e78907f7505","Type":"ContainerStarted","Data":"7462b8c307b88cb0e91062e72ade10b7d8fcb81254a12d475cfabe6a0ee8f921"} Nov 03 06:09:12.030585 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.030526 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podStartSLOduration=4.030507532 podStartE2EDuration="4.030507532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:12.030199214 +0000 UTC m=+839.838586480" watchObservedRunningTime="2026-11-03 06:09:12.030507532 +0000 UTC m=+839.838894802" Nov 03 06:09:12.083916 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.083852 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podStartSLOduration=3.083833118 podStartE2EDuration="3.083833118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:12.081369566 +0000 UTC m=+839.889756854" watchObservedRunningTime="2026-11-03 06:09:12.083833118 +0000 UTC m=+839.892220387" Nov 03 06:09:12.135998 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.135940 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podStartSLOduration=4.135898841 podStartE2EDuration="4.135898841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:12.129344544 +0000 UTC m=+839.937731813" watchObservedRunningTime="2026-11-03 06:09:12.135898841 +0000 UTC m=+839.944286111" Nov 03 06:09:12.274632 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.274537 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:12.275801 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.275767 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 06:09:12.275946 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.275818 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 06:09:12.786483 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.786451 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:12.812077 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.787033 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:12.846436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.835668 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 06:09:12.846436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.835732 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 06:09:12.846436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.836964 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 06:09:12.846436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:12.837014 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 06:09:13.207914 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.207876 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" event={"ID":"ff1bc896-f654-4773-9226-b5b099bb604b","Type":"ContainerStarted","Data":"2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd"} Nov 03 06:09:13.299412 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.297175 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 06:09:13.299412 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.297244 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 06:09:13.334973 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.321904 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podStartSLOduration=4.321880174 podStartE2EDuration="4.321880174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:13.30698735 +0000 UTC m=+841.115374621" watchObservedRunningTime="2026-11-03 06:09:13.321880174 +0000 UTC m=+841.130267442" Nov 03 06:09:13.368952 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.368920 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 06:09:13.369199 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.369175 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 06:09:13.370416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.370154 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:13.653534 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.653484 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:13.654261 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.654226 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:13.738587 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.738538 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 06:09:13.738884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.738842 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 06:09:13.739640 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.739571 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 06:09:13.739801 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.739778 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 06:09:13.789105 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.789067 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 06:09:13.823462 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.790173 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:09:13.836465 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.790202 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:09:13.928635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.928518 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:09:14.003416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.989567 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:14.003416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:13.991479 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:14.089116 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.089067 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:14.278352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.278260 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 06:09:14.278605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.278576 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 06:09:14.436556 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.436520 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 06:09:14.454024 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.453977 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 06:09:14.609543 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.609491 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:14.662410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.660565 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 06:09:14.662410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.661480 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:09:14.740524 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.739763 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 06:09:14.740524 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.739851 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 06:09:14.740524 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.740430 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 06:09:14.740524 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.740466 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 06:09:14.955469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.955172 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:09:15.023870 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.956576 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:15.023870 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.981527 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:09:15.023870 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.981591 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:09:15.023870 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:14.997462 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:15.044467 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.024204 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:09:15.103410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.059364 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:09:15.103410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.059452 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:09:15.103410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.060833 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:15.103410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.075283 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:09:15.103410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.076110 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:09:15.419418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.385021 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 06:09:15.419418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.397779 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 06:09:15.508524 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.466005 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Nov 03 06:09:15.508524 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.483534 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 06:09:15.508524 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.483593 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 06:09:15.650103 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.614817 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:15.650103 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.615127 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:09:15.650103 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.630841 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:15.689669 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.689544 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:15.769114 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.767509 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:15.835719 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.835670 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 06:09:15.836069 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.835951 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 06:09:15.840635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.840601 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 06:09:15.840831 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.840807 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 06:09:15.961467 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.955952 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:16.016415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:15.969080 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:09:16.041417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.026772 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:16.041417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.030682 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:16.123890 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.067967 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:16.123890 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.093733 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:16.134235 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.134204 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:09:16.307543 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.307500 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 06:09:16.307818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.307792 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 06:09:16.411001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.410906 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 06:09:16.411270 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.411242 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 06:09:16.482439 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.475000 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:09:16.513702 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.513674 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:09:16.628655 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.628550 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:16.652323 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.652280 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:16.661135 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.661105 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:09:16.752590 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.716481 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 06:09:16.798750 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.769790 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:16.804549 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.804523 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:09:16.900499 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.883474 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 06:09:16.900499 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.883542 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 06:09:16.900499 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.883631 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 06:09:16.900499 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:16.883653 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 06:09:17.017528 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.017491 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:09:17.105415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.101234 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:17.109024 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.108982 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:17.273607 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.268956 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:09:17.298367 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.298322 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 06:09:17.353436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.299653 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 06:09:17.388418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.364146 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 06:09:17.388418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.364206 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 06:09:17.625414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.594564 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:17.689428 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.643782 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:17.733422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.684741 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:17.733790 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.733750 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 06:09:17.737795 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.737764 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 06:09:17.738028 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.737983 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 06:09:17.751893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.738496 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 06:09:17.752154 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.752109 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 06:09:17.818722 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:17.818671 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:18.026505 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.020590 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:18.170890 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.136422 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:09:18.355903 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.281534 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:09:18.355903 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.286498 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:18.361934 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.361875 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:09:18.397496 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.393632 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 06:09:18.397496 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.393719 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 06:09:18.397496 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.397097 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 06:09:18.397496 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.397149 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 06:09:18.604414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.602031 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:18.806447 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.803934 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 06:09:18.806447 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.805028 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 06:09:18.851452 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.848065 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:18.887085 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.882571 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Nov 03 06:09:18.925535 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:18.846630 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:19.042775 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.027458 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 06:09:19.122527 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.103741 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:19.126233 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.126031 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 06:09:19.126498 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.126463 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 06:09:19.326420 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.315684 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:19.339424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.331910 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 06:09:19.410472 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.407446 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 06:09:19.410472 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:09:19.377282 4855 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.02s" Nov 03 06:09:19.410472 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.402217 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:09:19.410472 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.402874 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:09:19.410472 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.402848 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:09:19.410472 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.407880 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:09:19.418451 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.408861 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:09:19.518643 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.432706 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 06:09:19.518643 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.432766 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 06:09:19.522918 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.522890 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:09:19.609411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.608483 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:09:19.825796 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.825478 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 06:09:19.825796 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.825532 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 06:09:19.825796 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.825692 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 06:09:19.825796 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.825740 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 06:09:19.886526 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.886477 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:19.931031 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.930990 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:19.950091 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:19.949056 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:20.058553 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.038571 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:20.110154 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.110020 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:20.126282 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.124713 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:09:20.282615 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.282574 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:09:20.295697 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.284466 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 06:09:20.295697 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.284519 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 06:09:20.307624 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.307597 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:09:20.340657 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.340597 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:09:20.381490 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.361955 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 06:09:20.381490 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.362016 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 06:09:20.433893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.433660 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:09:20.475655 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.475599 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:20.612895 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.612848 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:09:20.620408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.615572 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:09:20.620408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.615680 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:09:20.680566 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.678331 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:09:20.749915 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.749881 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 06:09:20.750163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.750130 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 06:09:20.750848 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.750827 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:09:20.754001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.753662 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 06:09:20.754001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.753702 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 06:09:20.769342 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.769316 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:09:20.851801 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:20.851776 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:09:21.374265 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:21.356448 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 06:09:21.374265 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:21.356511 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 06:09:21.603442 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:21.602171 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:21.658468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:21.654915 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:21.705243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:21.702019 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:09:22.057650 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:22.057619 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:22.059119 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:22.059098 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:22.061088 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:22.061073 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:09:22.423449 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:22.423329 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:22.425016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:22.424989 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:22.426930 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:22.426026 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:09:22.457828 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:22.457779 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:22.458422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:22.458400 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:22.459706 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:22.459686 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:09:23.925027 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:23.924943 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n"] Nov 03 06:09:23.931593 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:23.931565 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:23.957915 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:23.957885 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n"] Nov 03 06:09:24.078834 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.078791 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.078995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.078846 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.078995 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.078962 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43948c14-233c-4679-93bc-b73c2f09725f-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.079112 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.079027 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43948c14-233c-4679-93bc-b73c2f09725f-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.079211 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.079186 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.079266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.079238 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.079336 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.079315 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.079469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.079454 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6pw5\" (UniqueName: \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-kube-api-access-t6pw5\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.079539 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.079485 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.180974 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.180894 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.180974 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.180942 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.180980 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43948c14-233c-4679-93bc-b73c2f09725f-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181047 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43948c14-233c-4679-93bc-b73c2f09725f-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181134 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181170 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181215 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181333 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6pw5\" (UniqueName: \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-kube-api-access-t6pw5\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181358 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181372 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181421 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181722 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181534 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181722 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181616 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.181877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.181856 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43948c14-233c-4679-93bc-b73c2f09725f-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.183521 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.183496 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.183776 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.183760 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43948c14-233c-4679-93bc-b73c2f09725f-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.205944 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.205919 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.206149 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.206127 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6pw5\" (UniqueName: \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-kube-api-access-t6pw5\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.288266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.288233 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:24.389451 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.359849 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs"] Nov 03 06:09:24.389451 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.366030 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.444048 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.443883 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs"] Nov 03 06:09:24.524445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.516446 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.524445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.516511 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.524445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.516554 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.524445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.516735 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkz8\" (UniqueName: \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-kube-api-access-qfkz8\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.524445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.516783 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.524445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.516859 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.524445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.516921 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.524445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.517119 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.524445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.517265 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.593893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.593859 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf"] Nov 03 06:09:24.598296 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.598270 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.618545 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618509 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.618698 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618559 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.618698 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618629 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.618698 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618661 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.618875 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618703 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.618875 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618740 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.618875 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618794 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.618875 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618840 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.618875 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618869 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.619110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618904 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.619110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618952 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d48278d0-aa0f-4f79-af81-39f95b6c5495-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.619110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.618977 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.619110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.619024 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.619110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.619093 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkz8\" (UniqueName: \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-kube-api-access-qfkz8\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.619367 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.619143 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.619367 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.619183 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhvw\" (UniqueName: \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-kube-api-access-dwhvw\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.619367 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.619217 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.619367 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.619247 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.619743 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.619667 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.621086 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.621061 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.621329 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.621309 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.621607 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.621586 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.622446 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.622385 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.623325 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.623308 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.624815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.624735 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n"] Nov 03 06:09:24.626444 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.626377 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.627199 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.627173 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:24.627273 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.627230 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:24.627273 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.627259 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:24.630084 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.630061 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf"] Nov 03 06:09:24.647890 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.647861 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.650505 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.650480 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkz8\" (UniqueName: \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-kube-api-access-qfkz8\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.721023 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.720991 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.721023 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721027 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.721274 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721112 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.721274 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721157 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.721274 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721193 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.721565 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721322 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d48278d0-aa0f-4f79-af81-39f95b6c5495-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.721565 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721347 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.721565 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721410 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.721565 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721511 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhvw\" (UniqueName: \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-kube-api-access-dwhvw\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.721565 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721549 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.721760 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721618 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.724266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.721938 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.724266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.722079 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.724266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.722342 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d48278d0-aa0f-4f79-af81-39f95b6c5495-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.724266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.723982 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.724266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.724219 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.748963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.748935 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.749155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.749131 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhvw\" (UniqueName: \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-kube-api-access-dwhvw\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:24.789700 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.789666 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:24.917945 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:24.917911 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:25.124749 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.123543 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs"] Nov 03 06:09:25.128068 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.125488 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:25.128068 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.125625 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:25.128068 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.125725 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:25.260027 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.259931 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf"] Nov 03 06:09:25.284321 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:09:25.284289 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48278d0_aa0f_4f79_af81_39f95b6c5495.slice/crio-93ffe2fa749ff4a4752abd6549132e2d7b4b1c8787eca1872e4016c4da078eb7 WatchSource:0}: Error finding container 93ffe2fa749ff4a4752abd6549132e2d7b4b1c8787eca1872e4016c4da078eb7: Status 404 returned error can't find the container with id 93ffe2fa749ff4a4752abd6549132e2d7b4b1c8787eca1872e4016c4da078eb7 Nov 03 06:09:25.288694 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.288654 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:25.288793 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.288745 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:25.288850 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.288789 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:09:25.372613 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.372580 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" event={"ID":"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c","Type":"ContainerStarted","Data":"f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af"} Nov 03 06:09:25.372788 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.372776 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" event={"ID":"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c","Type":"ContainerStarted","Data":"e34fd5cd144718ee3cea995f9cbed5e63d6edd0c250b1916934d1f2fc840117e"} Nov 03 06:09:25.374770 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.374736 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" event={"ID":"43948c14-233c-4679-93bc-b73c2f09725f","Type":"ContainerStarted","Data":"7d20f21835eee29c53b5ffda686806674f84e6264eb08f88bbb34c6a7fdd7a62"} Nov 03 06:09:25.374914 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.374814 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" event={"ID":"43948c14-233c-4679-93bc-b73c2f09725f","Type":"ContainerStarted","Data":"57db912b201882eaa3cc09a3184dfc800bd780843aa87056b9b92c78675ef0cd"} Nov 03 06:09:25.375980 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.375957 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" event={"ID":"d48278d0-aa0f-4f79-af81-39f95b6c5495","Type":"ContainerStarted","Data":"93ffe2fa749ff4a4752abd6549132e2d7b4b1c8787eca1872e4016c4da078eb7"} Nov 03 06:09:25.422781 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.422733 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podStartSLOduration=1.422718447 podStartE2EDuration="1.422718447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:25.418704032 +0000 UTC m=+853.227091301" watchObservedRunningTime="2026-11-03 06:09:25.422718447 +0000 UTC m=+853.231105711" Nov 03 06:09:25.472965 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.472914 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" podStartSLOduration=2.472897627 podStartE2EDuration="2.472897627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:25.472708697 +0000 UTC m=+853.281095965" watchObservedRunningTime="2026-11-03 06:09:25.472897627 +0000 UTC m=+853.281284894" Nov 03 06:09:25.790535 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:25.790500 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:26.298795 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:26.298758 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:26.312609 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:26.312568 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 06:09:26.312900 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:26.312869 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 06:09:26.489446 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:26.478511 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" event={"ID":"d48278d0-aa0f-4f79-af81-39f95b6c5495","Type":"ContainerStarted","Data":"b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd"} Nov 03 06:09:26.736464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:26.715968 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" podStartSLOduration=2.715947837 podStartE2EDuration="2.715947837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:26.695868479 +0000 UTC m=+854.504255748" watchObservedRunningTime="2026-11-03 06:09:26.715947837 +0000 UTC m=+854.524335107" Nov 03 06:09:26.792706 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:26.792670 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 06:09:26.792952 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:26.792921 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 06:09:26.923165 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:26.923135 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:26.926300 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:26.924509 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 06:09:26.926300 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:26.924562 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 06:09:27.302501 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:27.294273 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 06:09:27.302501 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:27.294338 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 06:09:27.797470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:27.797176 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 06:09:27.797470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:27.797255 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 06:09:27.919077 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:27.919043 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 06:09:27.919243 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:27.919093 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 06:09:28.304469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:28.295922 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 06:09:28.304469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:28.295976 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 06:09:28.790875 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:28.790781 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 06:09:28.791124 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:28.791095 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 06:09:28.932415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:28.922009 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 06:09:28.932415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:28.922086 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 06:09:29.093415 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:09:29.086681 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:09:29.288773 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:29.288735 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 06:09:29.289075 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:29.289044 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 06:09:29.794413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:29.793782 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 06:09:29.794413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:29.793848 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 06:09:29.918634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:29.918601 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 06:09:29.918800 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:29.918657 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 06:09:30.288970 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:30.288932 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 06:09:30.290531 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:30.290497 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 06:09:30.797841 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:30.797799 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 06:09:30.798483 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:30.798450 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 06:09:30.918883 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:30.918849 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 06:09:30.919078 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:30.918909 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 06:09:31.315472 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:31.293619 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 06:09:31.315765 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:31.315728 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 06:09:31.791017 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:31.790925 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 06:09:31.791017 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:31.790992 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 06:09:31.918547 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:31.918507 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 06:09:31.918953 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:31.918591 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 06:09:32.791095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:32.791054 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 06:09:32.791278 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:32.791123 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 06:09:32.794695 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:32.794665 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:32.795837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:32.795813 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:32.796975 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:32.796946 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:09:33.549778 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:33.549736 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:33.550619 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:33.550589 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:33.551532 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:33.551503 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:09:33.795494 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:33.795461 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:33.795776 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:33.795753 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:33.796549 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:33.796526 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:09:40.496916 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:40.496883 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:09:42.594418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:42.590528 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Nov 03 06:09:42.600810 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:42.594491 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:42.632415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:42.627853 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:42.651425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:42.648998 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 06:09:42.863468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:42.849783 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:43.031415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.024963 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:43.054566 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.054510 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:09:43.284139 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.179260 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l"] Nov 03 06:09:43.315190 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.289733 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd"] Nov 03 06:09:43.442414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.441947 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded" Nov 03 06:09:43.480034 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.479995 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:43.599930 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.599888 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:43.601084 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.601060 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:09:43.716484 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.679707 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:43.746128 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.746098 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:09:43.747279 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.679756 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:43.773414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.748711 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:09:43.812412 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.807839 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:43.865647 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.865546 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Nov 03 06:09:43.866620 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:43.866596 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:09:44.064417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.038946 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:09:44.096415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.066939 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:09:44.117468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.094572 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:44.183490 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.096364 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Nov 03 06:09:44.229416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.204294 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:44.836451 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.205562 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:09:44.855803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.855743 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 06:09:44.881633 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.585689 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:44.956659 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.956606 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:44.990817 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.300120 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:44.992425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.747360 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:44.992693 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.753151 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 06:09:45.003811 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:44.645537 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:45.085454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:45.069457 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:09:45.352521 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:45.332487 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:09:45.620626 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:45.620528 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:45.824577 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:45.824523 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:45.871128 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:45.871004 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:45.889607 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:45.889570 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:45.966434 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:45.957328 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:45.994441 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:45.993406 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:46.020827 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:45.994714 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:09:46.020827 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:45.994942 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:09:46.020827 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:46.005709 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:46.126180 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:46.126079 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:46.179453 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:09:46.154157 4855 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.317s" Nov 03 06:09:46.182318 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:46.182293 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:09:46.182554 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:46.182539 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:09:46.182685 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:46.182673 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:09:46.182805 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:46.182793 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:09:46.381596 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:46.381488 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:46.759625 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:46.759509 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:47.204464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.027435 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 06:09:47.204464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.054502 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:47.204464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.055614 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:47.204464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.068479 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:09:47.204464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.081675 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:09:47.204464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.082808 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:09:47.204464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.082848 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 06:09:47.204464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.082874 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Nov 03 06:09:47.204464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.133950 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:09:47.470874 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.470762 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:09:47.578415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.559034 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:09:47.578415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:47.559102 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:48.184513 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.184477 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:48.185012 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.184985 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 06:09:48.185277 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.185244 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:09:48.185521 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.185500 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 06:09:48.185853 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.185832 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:09:48.186182 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.186145 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:09:48.186518 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.186497 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:09:48.186887 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.186862 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 06:09:48.187668 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.187646 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:48.209679 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.209647 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:09:48.210278 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.210231 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:48.447629 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.447504 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:09:48.447893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.447861 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:48.490186 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.490147 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:09:48.490481 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.490450 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:48.490725 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.490701 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:48.802803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.802509 4855 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:09:48.802803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:48.802565 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:49.191584 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.187637 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:49.191584 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.187795 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Nov 03 06:09:49.193042 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.192952 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:49.196119 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.196075 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:49.196819 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.196795 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 06:09:49.197293 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.197269 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:09:49.197843 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.197815 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 06:09:49.198357 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.198336 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:09:49.249494 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.229851 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:09:49.250018 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.231538 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:09:49.297322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.297278 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:09:49.298209 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.298183 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:09:49.300757 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.300737 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:09:49.300984 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.300971 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:09:49.301794 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.301772 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:09:49.320906 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.320880 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:09:49.346600 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.346575 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:09:49.353687 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.353666 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:09:49.355440 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.355198 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm"] Nov 03 06:09:49.365661 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.365643 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:09:49.371709 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:49.371686 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:09:54.355852 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:09:54.355809 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" podUID="536311c9-35e9-4309-a39a-aa592e4e3112" containerName="payload-processing" containerID="cri-o://acda9f71ce6486797dcdc558701e986441597807a6a587696c7c3ce458cfb428" gracePeriod=25 Nov 03 06:09:58.897470 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:09:58.897426 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:10:03.686206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:03.686163 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" podUID="536311c9-35e9-4309-a39a-aa592e4e3112" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 06:10:04.111004 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.110923 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" containerID="cri-o://a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc" gracePeriod=40 Nov 03 06:10:04.355784 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.355743 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" containerID="cri-o://40554fc8e4d31302e67c21eb923e9735bcfff2190b5d001d1cbb0faaedf3f773" gracePeriod=39 Nov 03 06:10:04.510008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.509963 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:10:04.540857 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.540826 4855 generic.go:356] "Generic (PLEG): container finished" podID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerID="a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc" exitCode=0 Nov 03 06:10:04.541016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.540891 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" Nov 03 06:10:04.541016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.540904 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" event={"ID":"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc","Type":"ContainerDied","Data":"a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc"} Nov 03 06:10:04.541016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.540952 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l" event={"ID":"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc","Type":"ContainerDied","Data":"d78158cfd402faebcdd34bf653fb4e02431dd6c735c1ead0d7138f4e210c71ac"} Nov 03 06:10:04.541016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.540977 4855 scope.go:122] "RemoveContainer" containerID="a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc" Nov 03 06:10:04.543734 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.543702 4855 generic.go:356] "Generic (PLEG): container finished" podID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerID="40554fc8e4d31302e67c21eb923e9735bcfff2190b5d001d1cbb0faaedf3f773" exitCode=0 Nov 03 06:10:04.543864 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.543781 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" event={"ID":"c2c59a2e-54d8-497a-a4ce-37039429fde0","Type":"ContainerDied","Data":"40554fc8e4d31302e67c21eb923e9735bcfff2190b5d001d1cbb0faaedf3f773"} Nov 03 06:10:04.557616 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.557589 4855 scope.go:122] "RemoveContainer" containerID="a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc" Nov 03 06:10:04.558061 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:04.557987 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc\": container with ID starting with a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc not found: ID does not exist" containerID="a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc" Nov 03 06:10:04.558061 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.558027 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc"} err="failed to get container status \"a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc\": rpc error: code = NotFound desc = could not find container \"a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc\": container with ID starting with a70afa4628f19275830243574dd544df7903e0dff0c1d603ba4b67bedc862cbc not found: ID does not exist" Nov 03 06:10:04.610572 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.610531 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-kube-api-access-62xpq\" (UniqueName: \"kubernetes.io/projected/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-kube-api-access-62xpq\") pod \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " Nov 03 06:10:04.610761 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.610717 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tls-certs\") pod \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " Nov 03 06:10:04.610859 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.610845 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-dshm\") pod \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " Nov 03 06:10:04.610920 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.610875 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-model-cache\") pod \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " Nov 03 06:10:04.610960 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.610925 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tmp-dir\") pod \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " Nov 03 06:10:04.611002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.610959 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-home\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-home\") pod \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\" (UID: \"14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc\") " Nov 03 06:10:04.611880 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.611851 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-home" pod "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" (UID: "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:04.612732 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.612700 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-model-cache" pod "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" (UID: "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:04.613313 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.613070 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tmp-dir" pod "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" (UID: "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:04.616512 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.616486 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-kube-api-access-62xpq" pod "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" (UID: "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc"). InnerVolumeSpecName "kube-api-access-62xpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:04.618283 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.618260 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-dshm" pod "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" (UID: "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:04.618911 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.618873 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tls-certs" pod "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" (UID: "14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:10:04.650004 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.649976 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:10:04.713120 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713090 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c59a2e-54d8-497a-a4ce-37039429fde0-kube-api-access-n25gk\" (UniqueName: \"kubernetes.io/projected/c2c59a2e-54d8-497a-a4ce-37039429fde0-kube-api-access-n25gk\") pod \"c2c59a2e-54d8-497a-a4ce-37039429fde0\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " Nov 03 06:10:04.713666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713184 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-model-cache\") pod \"c2c59a2e-54d8-497a-a4ce-37039429fde0\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " Nov 03 06:10:04.713666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713267 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c59a2e-54d8-497a-a4ce-37039429fde0-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c59a2e-54d8-497a-a4ce-37039429fde0-tls-certs\") pod \"c2c59a2e-54d8-497a-a4ce-37039429fde0\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " Nov 03 06:10:04.713666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713318 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-dshm\") pod \"c2c59a2e-54d8-497a-a4ce-37039429fde0\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " Nov 03 06:10:04.713666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713360 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-tmp-dir\") pod \"c2c59a2e-54d8-497a-a4ce-37039429fde0\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " Nov 03 06:10:04.713666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713425 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-home\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-home\") pod \"c2c59a2e-54d8-497a-a4ce-37039429fde0\" (UID: \"c2c59a2e-54d8-497a-a4ce-37039429fde0\") " Nov 03 06:10:04.713938 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713905 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62xpq\" (UniqueName: \"kubernetes.io/projected/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-kube-api-access-62xpq\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.713938 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713917 4855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tls-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.713938 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713926 4855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-dshm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.713938 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713934 4855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-model-cache\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.714141 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713943 4855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-tmp-dir\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.714141 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.713950 4855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc-home\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.714241 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.714146 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-home" pod "c2c59a2e-54d8-497a-a4ce-37039429fde0" (UID: "c2c59a2e-54d8-497a-a4ce-37039429fde0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:04.714694 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.714559 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-model-cache" pod "c2c59a2e-54d8-497a-a4ce-37039429fde0" (UID: "c2c59a2e-54d8-497a-a4ce-37039429fde0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:04.714855 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.714816 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-tmp-dir" pod "c2c59a2e-54d8-497a-a4ce-37039429fde0" (UID: "c2c59a2e-54d8-497a-a4ce-37039429fde0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:04.717081 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.717050 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-dshm" pod "c2c59a2e-54d8-497a-a4ce-37039429fde0" (UID: "c2c59a2e-54d8-497a-a4ce-37039429fde0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:04.717305 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.717283 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c59a2e-54d8-497a-a4ce-37039429fde0-tls-certs" pod "c2c59a2e-54d8-497a-a4ce-37039429fde0" (UID: "c2c59a2e-54d8-497a-a4ce-37039429fde0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:10:04.719727 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.718784 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c59a2e-54d8-497a-a4ce-37039429fde0-kube-api-access-n25gk" pod "c2c59a2e-54d8-497a-a4ce-37039429fde0" (UID: "c2c59a2e-54d8-497a-a4ce-37039429fde0"). InnerVolumeSpecName "kube-api-access-n25gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:04.816016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.815968 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n25gk\" (UniqueName: \"kubernetes.io/projected/c2c59a2e-54d8-497a-a4ce-37039429fde0-kube-api-access-n25gk\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.816016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.816009 4855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-model-cache\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.816016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.816024 4855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c59a2e-54d8-497a-a4ce-37039429fde0-tls-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.816272 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.816036 4855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-dshm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.816272 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.816050 4855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-tmp-dir\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.816272 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.816062 4855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c59a2e-54d8-497a-a4ce-37039429fde0-home\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:04.890017 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.889922 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l"] Nov 03 06:10:04.893377 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:04.893353 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-2-w3-b7e447-kserve-54cf4469db-dp92l"] Nov 03 06:10:05.553031 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:05.552998 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" Nov 03 06:10:05.553207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:05.553025 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd" event={"ID":"c2c59a2e-54d8-497a-a4ce-37039429fde0","Type":"ContainerDied","Data":"e893ab16225e80bd21ac3874f96030323689756cde513479cf3f8e0a231cb65d"} Nov 03 06:10:05.553207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:05.553080 4855 scope.go:122] "RemoveContainer" containerID="40554fc8e4d31302e67c21eb923e9735bcfff2190b5d001d1cbb0faaedf3f773" Nov 03 06:10:05.610256 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:05.610175 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd"] Nov 03 06:10:05.623608 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:05.623577 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/distinct-w3-b7e447-kserve-c8c66c745-6vpdd"] Nov 03 06:10:06.334050 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:06.334016 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" path="/var/lib/kubelet/pods/14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc/volumes" Nov 03 06:10:06.334557 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:06.334542 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" path="/var/lib/kubelet/pods/c2c59a2e-54d8-497a-a4ce-37039429fde0/volumes" Nov 03 06:10:09.174048 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.174010 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw"] Nov 03 06:10:09.174864 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.174841 4855 cpu_manager.go:414] "containerMap: removing container" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" Nov 03 06:10:09.174864 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.174864 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" Nov 03 06:10:09.175048 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.174904 4855 cpu_manager.go:414] "containerMap: removing container" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" Nov 03 06:10:09.175048 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.174912 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" Nov 03 06:10:09.175142 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.175116 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b4bbcb-54da-4bd5-a7ed-7e1e276a58bc" containerName="main" Nov 03 06:10:09.175142 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.175137 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c59a2e-54d8-497a-a4ce-37039429fde0" containerName="main" Nov 03 06:10:09.179924 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.179902 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.205411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.204133 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw"] Nov 03 06:10:09.318078 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.318038 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.318285 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.318093 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.318285 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.318250 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.318423 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.318307 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5jc\" (UniqueName: \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-kube-api-access-pp5jc\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.318478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.318456 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.318539 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.318495 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7ea471-1f77-49c0-92dc-139f56db1d14-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.318634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.318614 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.318695 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.318641 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.318695 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.318678 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.419745 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.419709 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.419904 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.419761 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7ea471-1f77-49c0-92dc-139f56db1d14-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420150 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420005 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420150 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420040 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420150 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420096 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420419 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420200 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420419 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420308 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420419 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420340 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420642 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420514 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420642 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420573 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5jc\" (UniqueName: \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-kube-api-access-pp5jc\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420785 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420761 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420847 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420769 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7ea471-1f77-49c0-92dc-139f56db1d14-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.420957 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.420935 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.421195 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.421176 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.422439 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.422417 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.422620 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.422602 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.444694 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.444641 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5jc\" (UniqueName: \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-kube-api-access-pp5jc\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.446216 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.446195 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.503885 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.503842 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:09.846514 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.846484 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw"] Nov 03 06:10:09.846961 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:10:09.846938 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7ea471_1f77_49c0_92dc_139f56db1d14.slice/crio-8b2fabc501cddae8de27712cc7982ca1f4d7c67ac0fa44c5a29a71824a70386f WatchSource:0}: Error finding container 8b2fabc501cddae8de27712cc7982ca1f4d7c67ac0fa44c5a29a71824a70386f: Status 404 returned error can't find the container with id 8b2fabc501cddae8de27712cc7982ca1f4d7c67ac0fa44c5a29a71824a70386f Nov 03 06:10:09.850180 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.850140 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:09.850298 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.850215 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:09.850298 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:09.850250 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:10.593589 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:10.593553 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" event={"ID":"8f7ea471-1f77-49c0-92dc-139f56db1d14","Type":"ContainerStarted","Data":"55491e7918d0c8346bfafcff110bb85b41026bbe975dcbc5bd01054ea3a5939f"} Nov 03 06:10:10.593983 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:10.593610 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" event={"ID":"8f7ea471-1f77-49c0-92dc-139f56db1d14","Type":"ContainerStarted","Data":"8b2fabc501cddae8de27712cc7982ca1f4d7c67ac0fa44c5a29a71824a70386f"} Nov 03 06:10:10.669795 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:10.669742 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podStartSLOduration=1.669728427 podStartE2EDuration="1.669728427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:10.654938412 +0000 UTC m=+898.463325718" watchObservedRunningTime="2026-11-03 06:10:10.669728427 +0000 UTC m=+898.478115694" Nov 03 06:10:11.509409 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:11.509328 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:11.509734 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:11.509484 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:11.509734 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:11.509544 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:11.784573 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:11.784478 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv"] Nov 03 06:10:12.428592 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:12.428566 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:10:12.457797 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:12.457599 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:10:12.504766 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:12.504737 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:12.504916 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:12.504790 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:13.505045 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:13.505005 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:13.505596 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:13.505072 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:13.731123 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:13.731083 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" podUID="536311c9-35e9-4309-a39a-aa592e4e3112" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 06:10:14.535259 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:14.507785 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:14.535259 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:14.507850 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:14.709491 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:14.697251 4855 generic.go:356] "Generic (PLEG): container finished" podID="536311c9-35e9-4309-a39a-aa592e4e3112" containerID="acda9f71ce6486797dcdc558701e986441597807a6a587696c7c3ce458cfb428" exitCode=0 Nov 03 06:10:14.709491 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:14.697540 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" event={"ID":"536311c9-35e9-4309-a39a-aa592e4e3112","Type":"ContainerDied","Data":"acda9f71ce6486797dcdc558701e986441597807a6a587696c7c3ce458cfb428"} Nov 03 06:10:15.073567 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:15.073512 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:15.106606 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:15.106566 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:15.177634 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:15.177570 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:15.195421 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:15.191504 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:15.195421 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:15.191919 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:10:15.195421 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:15.192190 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:10:15.195421 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:15.192465 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Nov 03 06:10:15.500475 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:15.494485 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:15.517424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:15.515848 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:15.517424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:15.515919 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:16.107550 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.107493 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:10:16.200414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.137692 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:10:16.200414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.196161 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:10:16.200414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.196222 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:10:16.200414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.178489 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 06:10:16.200414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.194662 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:16.200868 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.194699 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:10:16.211411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.194720 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:10:16.211411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.201283 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:10:16.211411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.201336 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:10:16.211411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.201370 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:10:16.211411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.201422 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:10:16.211411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.194739 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:10:16.211411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.203629 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:10:16.510438 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.510316 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:16.700414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.674596 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:17.302878 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.302841 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:17.327664 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.069278 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:10:17.369658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.219906 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:10:17.370240 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.370207 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" event={"ID":"536311c9-35e9-4309-a39a-aa592e4e3112","Type":"ContainerDied","Data":"5f2f0089a8a845dd5ba51aaac4ad5ab4fdbc29d84d52997e98e0ec891b2b3308"} Nov 03 06:10:17.432443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.382101 4855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2f0089a8a845dd5ba51aaac4ad5ab4fdbc29d84d52997e98e0ec891b2b3308" Nov 03 06:10:17.432443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.219984 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Nov 03 06:10:17.432443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.302780 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:10:17.432443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:16.743703 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 03 06:10:17.432443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.219942 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:10:17.432443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.372892 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:10:17.432443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.372922 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:10:17.432443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.372943 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:10:17.432443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.219964 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 06:10:17.508429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.497206 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536311c9-35e9-4309-a39a-aa592e4e3112-kube-api-access-rzps9\" (UniqueName: \"kubernetes.io/projected/536311c9-35e9-4309-a39a-aa592e4e3112-kube-api-access-rzps9\") pod \"536311c9-35e9-4309-a39a-aa592e4e3112\" (UID: \"536311c9-35e9-4309-a39a-aa592e4e3112\") " Nov 03 06:10:17.508429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.497312 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536311c9-35e9-4309-a39a-aa592e4e3112-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536311c9-35e9-4309-a39a-aa592e4e3112-plugins-config-volume\") pod \"536311c9-35e9-4309-a39a-aa592e4e3112\" (UID: \"536311c9-35e9-4309-a39a-aa592e4e3112\") " Nov 03 06:10:17.533424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.525951 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:10:17.561548 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.543720 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536311c9-35e9-4309-a39a-aa592e4e3112-plugins-config-volume" pod "536311c9-35e9-4309-a39a-aa592e4e3112" (UID: "536311c9-35e9-4309-a39a-aa592e4e3112"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:10:17.586415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.573555 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:17.586415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.573619 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:17.915470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.908993 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:17.915470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.909060 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:17.972411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:17.955732 4855 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536311c9-35e9-4309-a39a-aa592e4e3112-plugins-config-volume\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:18.239862 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.210582 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536311c9-35e9-4309-a39a-aa592e4e3112-kube-api-access-rzps9" pod "536311c9-35e9-4309-a39a-aa592e4e3112" (UID: "536311c9-35e9-4309-a39a-aa592e4e3112"). InnerVolumeSpecName "kube-api-access-rzps9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:18.240250 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.240129 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:18.254796 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.254755 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:18.305611 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.305571 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzps9\" (UniqueName: \"kubernetes.io/projected/536311c9-35e9-4309-a39a-aa592e4e3112-kube-api-access-rzps9\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:18.415610 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.402497 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:18.432718 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.432672 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Nov 03 06:10:18.583373 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.561781 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:18.591551 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.583687 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:18.710425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.685739 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:18.710425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.685818 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:18.710425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.687881 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:10:18.710425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.688558 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:18.710425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.689286 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:10:18.767415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.738705 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 06:10:18.893477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.868914 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:18.966963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:18.966904 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:10:19.015417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.015001 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:19.015417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.015065 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:19.104867 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.101191 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm" Nov 03 06:10:19.196476 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.189899 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Nov 03 06:10:19.406502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.403480 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:19.433980 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:19.433928 4855 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536311c9_35e9_4309_a39a_aa592e4e3112.slice\": RecentStats: unable to find data in memory cache]" Nov 03 06:10:19.451642 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.451483 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm"] Nov 03 06:10:19.477416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.468507 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b7e447-f4d488b7-cz5gm"] Nov 03 06:10:19.527420 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.509589 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:19.527771 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.527726 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:19.578416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.577961 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:19.734303 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.692536 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:19.734303 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.695036 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:19.734303 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.695415 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:19.734303 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.734128 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:19.734303 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.734184 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:19.803413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.798833 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:19.907443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.892519 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:19.907443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.892578 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:19.907443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.892870 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:19.907443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.892906 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:19.940416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:19.918600 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:20.039162 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.035667 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:10:20.176615 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.176565 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:20.176905 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.176856 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:20.391447 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.391353 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536311c9-35e9-4309-a39a-aa592e4e3112" path="/var/lib/kubelet/pods/536311c9-35e9-4309-a39a-aa592e4e3112/volumes" Nov 03 06:10:20.421592 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.421546 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:20.513196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.513160 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:20.513518 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.513473 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:20.579519 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.579468 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Nov 03 06:10:20.717474 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.692710 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:10:20.719316 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.719185 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:20.761415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.725258 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:10:20.793413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.761788 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:20.818412 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.804536 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 06:10:20.834445 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.834381 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:20.834757 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:20.834712 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:21.053477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.015472 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:10:21.117871 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.098671 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:21.136838 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.098724 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:21.162533 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.162487 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:21.431467 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.430503 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:21.532748 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.532709 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:21.533024 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.532994 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:21.625099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.625058 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:21.639426 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.630248 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:21.719597 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.719489 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:21.729511 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.729465 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:10:21.739580 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.739536 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:21.809316 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:21.809276 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Nov 03 06:10:22.104592 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.104533 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:22.162964 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.162923 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:22.518474 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.480174 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:10:22.605415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.565503 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:22.605415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.576127 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:22.628440 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.626033 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:22.628440 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.626375 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Nov 03 06:10:22.766474 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.743558 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:22.766474 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.743626 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:22.782483 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.771902 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:22.782483 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.773528 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:10:22.783123 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.783071 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:10:22.841420 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:22.825007 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:23.271718 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.260257 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:10:23.507425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.488596 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:23.507425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.501896 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 06:10:23.581578 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.581525 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:10:23.741883 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.741837 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:23.764416 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:23.744502 4855 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.402s" Nov 03 06:10:23.764416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.744542 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg"] Nov 03 06:10:23.764416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.744610 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl"] Nov 03 06:10:23.764416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.744640 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6"] Nov 03 06:10:23.764416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.744670 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj"] Nov 03 06:10:23.764416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.744690 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2"] Nov 03 06:10:23.764416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.745220 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" containerID="cri-o://c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65" gracePeriod=30 Nov 03 06:10:23.764416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.745450 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" containerID="cri-o://2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd" gracePeriod=30 Nov 03 06:10:23.764416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.745667 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" containerID="cri-o://734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf" gracePeriod=30 Nov 03 06:10:23.779293 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.778315 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:10:23.808480 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.808320 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 06:10:23.883708 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.869477 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 03 06:10:23.883708 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.880995 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:10:23.906125 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.906075 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:23.906439 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:23.906379 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:24.030416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.023455 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:10:24.085407 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.072403 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:10:24.114169 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.101225 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:10:24.120967 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.120844 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:10:24.121598 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.121580 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:10:24.173849 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.173779 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:10:24.177968 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.177931 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:10:24.504903 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.504820 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:24.505138 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.505116 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 06:10:24.505222 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.505170 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 06:10:24.885674 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:24.885636 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8"] Nov 03 06:10:26.544561 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:26.544524 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:10:26.545010 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:26.544590 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:26.806477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:26.806434 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" containerID="cri-o://b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040" gracePeriod=45 Nov 03 06:10:27.179209 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.179176 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:10:27.192630 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.192594 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:27.194189 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.194140 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:10:27.286981 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.286943 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f485f34b-569a-4647-b63f-3b1ce1f5b2df-kube-api-access-pjw28\" (UniqueName: \"kubernetes.io/projected/f485f34b-569a-4647-b63f-3b1ce1f5b2df-kube-api-access-pjw28\") pod \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " Nov 03 06:10:27.287266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.287057 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-home\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-home\") pod \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " Nov 03 06:10:27.287266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.287196 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tmp-dir\") pod \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " Nov 03 06:10:27.295053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.287292 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-dshm\") pod \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " Nov 03 06:10:27.295053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.287468 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-home" pod "f485f34b-569a-4647-b63f-3b1ce1f5b2df" (UID: "f485f34b-569a-4647-b63f-3b1ce1f5b2df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:27.295053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.287554 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tls-certs\" (UniqueName: \"kubernetes.io/secret/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tls-certs\") pod \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " Nov 03 06:10:27.295053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.287592 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-model-cache\") pod \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\" (UID: \"f485f34b-569a-4647-b63f-3b1ce1f5b2df\") " Nov 03 06:10:27.295053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.288614 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tmp-dir" pod "f485f34b-569a-4647-b63f-3b1ce1f5b2df" (UID: "f485f34b-569a-4647-b63f-3b1ce1f5b2df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:27.295053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.289569 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-model-cache" pod "f485f34b-569a-4647-b63f-3b1ce1f5b2df" (UID: "f485f34b-569a-4647-b63f-3b1ce1f5b2df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:27.295053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.291156 4855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-home\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:27.295053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.291286 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f485f34b-569a-4647-b63f-3b1ce1f5b2df-kube-api-access-pjw28" pod "f485f34b-569a-4647-b63f-3b1ce1f5b2df" (UID: "f485f34b-569a-4647-b63f-3b1ce1f5b2df"). InnerVolumeSpecName "kube-api-access-pjw28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:27.295483 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.291169 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-dshm" pod "f485f34b-569a-4647-b63f-3b1ce1f5b2df" (UID: "f485f34b-569a-4647-b63f-3b1ce1f5b2df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:27.298151 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.298119 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tls-certs" pod "f485f34b-569a-4647-b63f-3b1ce1f5b2df" (UID: "f485f34b-569a-4647-b63f-3b1ce1f5b2df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:10:27.392901 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.392814 4855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tmp-dir\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:27.392901 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.392851 4855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-dshm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:27.392901 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.392865 4855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f485f34b-569a-4647-b63f-3b1ce1f5b2df-tls-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:27.392901 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.392879 4855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f485f34b-569a-4647-b63f-3b1ce1f5b2df-model-cache\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:27.392901 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.392893 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjw28\" (UniqueName: \"kubernetes.io/projected/f485f34b-569a-4647-b63f-3b1ce1f5b2df-kube-api-access-pjw28\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:27.774670 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.774577 4855 generic.go:356] "Generic (PLEG): container finished" podID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerID="b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040" exitCode=0 Nov 03 06:10:27.774670 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.774622 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" Nov 03 06:10:27.775175 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.774669 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" event={"ID":"f485f34b-569a-4647-b63f-3b1ce1f5b2df","Type":"ContainerDied","Data":"b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040"} Nov 03 06:10:27.775175 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.774724 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv" event={"ID":"f485f34b-569a-4647-b63f-3b1ce1f5b2df","Type":"ContainerDied","Data":"a6ba590128ed03de47501796a9e3bb42715b7a8c1a6733498c92814e0d4448a3"} Nov 03 06:10:27.775175 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.774749 4855 scope.go:122] "RemoveContainer" containerID="b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040" Nov 03 06:10:27.784845 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.784827 4855 scope.go:122] "RemoveContainer" containerID="b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040" Nov 03 06:10:27.785130 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:27.785110 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040\": container with ID starting with b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040 not found: ID does not exist" containerID="b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040" Nov 03 06:10:27.785166 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.785140 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040"} err="failed to get container status \"b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040\": rpc error: code = NotFound desc = could not find container \"b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040\": container with ID starting with b7d5ccca6232048e151b94def388ed524fa312c85caed7f9ff4f7f26346fe040 not found: ID does not exist" Nov 03 06:10:27.833261 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.833236 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv"] Nov 03 06:10:27.841227 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:27.841196 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/premium-w5-ba6892-kserve-6bcd59c6b9-vjnkv"] Nov 03 06:10:28.337134 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:28.337093 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" path="/var/lib/kubelet/pods/f485f34b-569a-4647-b63f-3b1ce1f5b2df/volumes" Nov 03 06:10:28.886700 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:28.886667 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:10:29.055713 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.055688 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:10:29.124311 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.124283 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:10:29.233159 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233118 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-token\" (UniqueName: \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-token\") pod \"74571913-1a5a-4f8a-bf23-68211e1935d3\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " Nov 03 06:10:29.233352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233177 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-credential-socket\") pod \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " Nov 03 06:10:29.233352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233321 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-envoy\") pod \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " Nov 03 06:10:29.233490 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233436 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-token\" (UniqueName: \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-token\") pod \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " Nov 03 06:10:29.233549 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233509 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-data\") pod \"74571913-1a5a-4f8a-bf23-68211e1935d3\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " Nov 03 06:10:29.233603 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233580 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-podinfo\") pod \"74571913-1a5a-4f8a-bf23-68211e1935d3\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " Nov 03 06:10:29.233655 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233594 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-credential-socket" pod "beb9581f-e4cd-4cbd-b20e-4beeff24063c" (UID: "beb9581f-e4cd-4cbd-b20e-4beeff24063c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.233655 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233635 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-kube-api-access-slxw5\" (UniqueName: \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-kube-api-access-slxw5\") pod \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " Nov 03 06:10:29.233764 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233702 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-certs\") pod \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " Nov 03 06:10:29.233764 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233735 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-certs\") pod \"74571913-1a5a-4f8a-bf23-68211e1935d3\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " Nov 03 06:10:29.233867 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233790 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-socket\") pod \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " Nov 03 06:10:29.233954 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233912 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-kube-api-access-2vjf2\" (UniqueName: \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-kube-api-access-2vjf2\") pod \"74571913-1a5a-4f8a-bf23-68211e1935d3\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " Nov 03 06:10:29.234013 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233958 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74571913-1a5a-4f8a-bf23-68211e1935d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74571913-1a5a-4f8a-bf23-68211e1935d3-istiod-ca-cert\") pod \"74571913-1a5a-4f8a-bf23-68211e1935d3\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " Nov 03 06:10:29.234013 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.233997 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-envoy\") pod \"74571913-1a5a-4f8a-bf23-68211e1935d3\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " Nov 03 06:10:29.234112 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.234032 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-podinfo\") pod \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " Nov 03 06:10:29.234112 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.234087 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-credential-socket\") pod \"74571913-1a5a-4f8a-bf23-68211e1935d3\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " Nov 03 06:10:29.234215 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.234128 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-socket\") pod \"74571913-1a5a-4f8a-bf23-68211e1935d3\" (UID: \"74571913-1a5a-4f8a-bf23-68211e1935d3\") " Nov 03 06:10:29.234215 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.234182 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-data\") pod \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " Nov 03 06:10:29.234317 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.234224 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istiod-ca-cert\") pod \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\" (UID: \"beb9581f-e4cd-4cbd-b20e-4beeff24063c\") " Nov 03 06:10:29.234821 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.234802 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.235920 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.235621 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-token" pod "74571913-1a5a-4f8a-bf23-68211e1935d3" (UID: "74571913-1a5a-4f8a-bf23-68211e1935d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:29.235920 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.235751 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-socket" pod "74571913-1a5a-4f8a-bf23-68211e1935d3" (UID: "74571913-1a5a-4f8a-bf23-68211e1935d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.235920 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.235780 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istiod-ca-cert" pod "beb9581f-e4cd-4cbd-b20e-4beeff24063c" (UID: "beb9581f-e4cd-4cbd-b20e-4beeff24063c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:10:29.236129 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.235988 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-credential-socket" pod "74571913-1a5a-4f8a-bf23-68211e1935d3" (UID: "74571913-1a5a-4f8a-bf23-68211e1935d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.236129 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.236088 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-data" pod "beb9581f-e4cd-4cbd-b20e-4beeff24063c" (UID: "beb9581f-e4cd-4cbd-b20e-4beeff24063c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.236222 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.236205 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-certs" pod "74571913-1a5a-4f8a-bf23-68211e1935d3" (UID: "74571913-1a5a-4f8a-bf23-68211e1935d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.236720 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.236348 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-envoy" pod "beb9581f-e4cd-4cbd-b20e-4beeff24063c" (UID: "beb9581f-e4cd-4cbd-b20e-4beeff24063c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.236720 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.236355 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-data" pod "74571913-1a5a-4f8a-bf23-68211e1935d3" (UID: "74571913-1a5a-4f8a-bf23-68211e1935d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.236720 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.236430 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-certs" pod "beb9581f-e4cd-4cbd-b20e-4beeff24063c" (UID: "beb9581f-e4cd-4cbd-b20e-4beeff24063c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.236720 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.236591 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-socket" pod "beb9581f-e4cd-4cbd-b20e-4beeff24063c" (UID: "beb9581f-e4cd-4cbd-b20e-4beeff24063c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.236720 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.236685 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74571913-1a5a-4f8a-bf23-68211e1935d3-istiod-ca-cert" pod "74571913-1a5a-4f8a-bf23-68211e1935d3" (UID: "74571913-1a5a-4f8a-bf23-68211e1935d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:10:29.237729 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.237669 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-kube-api-access-2vjf2" pod "74571913-1a5a-4f8a-bf23-68211e1935d3" (UID: "74571913-1a5a-4f8a-bf23-68211e1935d3"). InnerVolumeSpecName "kube-api-access-2vjf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:29.238139 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.238107 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-token" pod "beb9581f-e4cd-4cbd-b20e-4beeff24063c" (UID: "beb9581f-e4cd-4cbd-b20e-4beeff24063c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:29.238532 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.238489 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-podinfo" pod "beb9581f-e4cd-4cbd-b20e-4beeff24063c" (UID: "beb9581f-e4cd-4cbd-b20e-4beeff24063c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:10:29.238803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.238770 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-kube-api-access-slxw5" pod "beb9581f-e4cd-4cbd-b20e-4beeff24063c" (UID: "beb9581f-e4cd-4cbd-b20e-4beeff24063c"). InnerVolumeSpecName "kube-api-access-slxw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:29.238935 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.238912 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-podinfo" pod "74571913-1a5a-4f8a-bf23-68211e1935d3" (UID: "74571913-1a5a-4f8a-bf23-68211e1935d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:10:29.240325 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.240292 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-envoy" pod "74571913-1a5a-4f8a-bf23-68211e1935d3" (UID: "74571913-1a5a-4f8a-bf23-68211e1935d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.254486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.254466 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:10:29.336240 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336202 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vjf2\" (UniqueName: \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-kube-api-access-2vjf2\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336240 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336238 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74571913-1a5a-4f8a-bf23-68211e1935d3-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336251 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336260 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336268 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336276 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336284 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336293 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336300 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336307 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336314 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336321 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336328 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74571913-1a5a-4f8a-bf23-68211e1935d3-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336336 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slxw5\" (UniqueName: \"kubernetes.io/projected/beb9581f-e4cd-4cbd-b20e-4beeff24063c-kube-api-access-slxw5\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336345 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336354 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74571913-1a5a-4f8a-bf23-68211e1935d3-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.336502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.336362 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb9581f-e4cd-4cbd-b20e-4beeff24063c-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.446473 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.443995 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-credential-socket\") pod \"ff1bc896-f654-4773-9226-b5b099bb604b\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " Nov 03 06:10:29.446473 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.444058 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-envoy\") pod \"ff1bc896-f654-4773-9226-b5b099bb604b\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " Nov 03 06:10:29.446473 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.444116 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1bc896-f654-4773-9226-b5b099bb604b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1bc896-f654-4773-9226-b5b099bb604b-istiod-ca-cert\") pod \"ff1bc896-f654-4773-9226-b5b099bb604b\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " Nov 03 06:10:29.446473 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.444153 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-istio-token\" (UniqueName: \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-istio-token\") pod \"ff1bc896-f654-4773-9226-b5b099bb604b\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " Nov 03 06:10:29.446473 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.444210 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff1bc896-f654-4773-9226-b5b099bb604b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1bc896-f654-4773-9226-b5b099bb604b-istio-podinfo\") pod \"ff1bc896-f654-4773-9226-b5b099bb604b\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " Nov 03 06:10:29.446473 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.444294 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-data\") pod \"ff1bc896-f654-4773-9226-b5b099bb604b\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " Nov 03 06:10:29.446473 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.444336 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-socket\") pod \"ff1bc896-f654-4773-9226-b5b099bb604b\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " Nov 03 06:10:29.467348 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.457800 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-data" pod "ff1bc896-f654-4773-9226-b5b099bb604b" (UID: "ff1bc896-f654-4773-9226-b5b099bb604b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.467348 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.458475 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-socket" pod "ff1bc896-f654-4773-9226-b5b099bb604b" (UID: "ff1bc896-f654-4773-9226-b5b099bb604b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.467348 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.464870 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-envoy" pod "ff1bc896-f654-4773-9226-b5b099bb604b" (UID: "ff1bc896-f654-4773-9226-b5b099bb604b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.467348 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.466166 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff1bc896-f654-4773-9226-b5b099bb604b-istio-podinfo" pod "ff1bc896-f654-4773-9226-b5b099bb604b" (UID: "ff1bc896-f654-4773-9226-b5b099bb604b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:10:29.468096 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.468049 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1bc896-f654-4773-9226-b5b099bb604b-istiod-ca-cert" pod "ff1bc896-f654-4773-9226-b5b099bb604b" (UID: "ff1bc896-f654-4773-9226-b5b099bb604b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:10:29.468875 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.468851 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-kube-api-access-9745r\" (UniqueName: \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-kube-api-access-9745r\") pod \"ff1bc896-f654-4773-9226-b5b099bb604b\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " Nov 03 06:10:29.469130 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.469066 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-certs\") pod \"ff1bc896-f654-4773-9226-b5b099bb604b\" (UID: \"ff1bc896-f654-4773-9226-b5b099bb604b\") " Nov 03 06:10:29.470073 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.470038 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-certs" pod "ff1bc896-f654-4773-9226-b5b099bb604b" (UID: "ff1bc896-f654-4773-9226-b5b099bb604b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.470880 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.470856 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.470982 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.470884 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.475019 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.474989 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-credential-socket" pod "ff1bc896-f654-4773-9226-b5b099bb604b" (UID: "ff1bc896-f654-4773-9226-b5b099bb604b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:29.475239 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.475214 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.475341 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.475329 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.475464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.475452 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1bc896-f654-4773-9226-b5b099bb604b-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.475563 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.475551 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1bc896-f654-4773-9226-b5b099bb604b-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.476578 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.476555 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-istio-token" pod "ff1bc896-f654-4773-9226-b5b099bb604b" (UID: "ff1bc896-f654-4773-9226-b5b099bb604b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:29.476879 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.476844 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-kube-api-access-9745r" pod "ff1bc896-f654-4773-9226-b5b099bb604b" (UID: "ff1bc896-f654-4773-9226-b5b099bb604b"). InnerVolumeSpecName "kube-api-access-9745r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:29.576680 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.576646 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9745r\" (UniqueName: \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-kube-api-access-9745r\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.576680 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.576677 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bc896-f654-4773-9226-b5b099bb604b-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.576680 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.576688 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1bc896-f654-4773-9226-b5b099bb604b-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:29.795117 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.795018 4855 generic.go:356] "Generic (PLEG): container finished" podID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerID="734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf" exitCode=0 Nov 03 06:10:29.795117 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.795072 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" Nov 03 06:10:29.795355 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.795115 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" event={"ID":"beb9581f-e4cd-4cbd-b20e-4beeff24063c","Type":"ContainerDied","Data":"734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf"} Nov 03 06:10:29.795355 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.795185 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6" event={"ID":"beb9581f-e4cd-4cbd-b20e-4beeff24063c","Type":"ContainerDied","Data":"913fd7c12db4d3101c340f3d7ba9fffb72d688bd257012cbcf12a3c7ab321580"} Nov 03 06:10:29.795355 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.795214 4855 scope.go:122] "RemoveContainer" containerID="734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf" Nov 03 06:10:29.796680 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.796652 4855 generic.go:356] "Generic (PLEG): container finished" podID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerID="c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65" exitCode=0 Nov 03 06:10:29.796809 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.796693 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" event={"ID":"74571913-1a5a-4f8a-bf23-68211e1935d3","Type":"ContainerDied","Data":"c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65"} Nov 03 06:10:29.796809 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.796707 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" Nov 03 06:10:29.796809 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.796747 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg" event={"ID":"74571913-1a5a-4f8a-bf23-68211e1935d3","Type":"ContainerDied","Data":"b6780409e4a02b4dbc90e51736e0fdb0b1ecfce89697a74023e10d242871a87a"} Nov 03 06:10:29.798310 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.798290 4855 generic.go:356] "Generic (PLEG): container finished" podID="ff1bc896-f654-4773-9226-b5b099bb604b" containerID="2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd" exitCode=0 Nov 03 06:10:29.798464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.798322 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" event={"ID":"ff1bc896-f654-4773-9226-b5b099bb604b","Type":"ContainerDied","Data":"2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd"} Nov 03 06:10:29.798464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.798352 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" event={"ID":"ff1bc896-f654-4773-9226-b5b099bb604b","Type":"ContainerDied","Data":"1807269992c355a72ab411451762eec99e6332052deaaa4f75a3ab167970396d"} Nov 03 06:10:29.798464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.798424 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl" Nov 03 06:10:29.827781 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.827754 4855 scope.go:122] "RemoveContainer" containerID="734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf" Nov 03 06:10:29.828175 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:29.828141 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf\": container with ID starting with 734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf not found: ID does not exist" containerID="734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf" Nov 03 06:10:29.828260 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.828188 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf"} err="failed to get container status \"734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf\": rpc error: code = NotFound desc = could not find container \"734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf\": container with ID starting with 734cb05cfbebf76cb45396ed527899270b6df9d77c22b04f0a04640cee05a7bf not found: ID does not exist" Nov 03 06:10:29.828260 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.828209 4855 scope.go:122] "RemoveContainer" containerID="c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65" Nov 03 06:10:29.838041 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.838021 4855 scope.go:122] "RemoveContainer" containerID="c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65" Nov 03 06:10:29.838325 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:29.838296 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65\": container with ID starting with c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65 not found: ID does not exist" containerID="c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65" Nov 03 06:10:29.838442 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.838334 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65"} err="failed to get container status \"c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65\": rpc error: code = NotFound desc = could not find container \"c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65\": container with ID starting with c8b0cd0b7b00a2e2fe40be56c176ac0d6b0f037a694432733e14f9e7a5c35d65 not found: ID does not exist" Nov 03 06:10:29.838442 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.838358 4855 scope.go:122] "RemoveContainer" containerID="2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd" Nov 03 06:10:29.848812 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.848791 4855 scope.go:122] "RemoveContainer" containerID="2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd" Nov 03 06:10:29.849106 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:29.849079 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd\": container with ID starting with 2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd not found: ID does not exist" containerID="2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd" Nov 03 06:10:29.849161 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.849116 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd"} err="failed to get container status \"2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd\": rpc error: code = NotFound desc = could not find container \"2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd\": container with ID starting with 2a3a3645cffc986926bb1e318623edbcbb632768f9fb1c9e79858a1c4f4edccd not found: ID does not exist" Nov 03 06:10:29.885780 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.885751 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6"] Nov 03 06:10:29.899147 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.899121 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-r9sp6"] Nov 03 06:10:29.926942 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.926916 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl"] Nov 03 06:10:29.930565 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.930539 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-lk8sl"] Nov 03 06:10:29.969272 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.969244 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg"] Nov 03 06:10:29.974342 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:29.974318 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-5d2zg"] Nov 03 06:10:30.334901 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:30.334868 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" path="/var/lib/kubelet/pods/74571913-1a5a-4f8a-bf23-68211e1935d3/volumes" Nov 03 06:10:30.335375 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:30.335361 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" path="/var/lib/kubelet/pods/beb9581f-e4cd-4cbd-b20e-4beeff24063c/volumes" Nov 03 06:10:30.335830 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:30.335816 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" path="/var/lib/kubelet/pods/ff1bc896-f654-4773-9226-b5b099bb604b/volumes" Nov 03 06:10:31.179204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:31.178464 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6"] Nov 03 06:10:36.179461 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:36.179402 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" podUID="97ba4903-d54a-4291-bd8b-811bc6b3302a" containerName="payload-pre-processing" containerID="cri-o://e4f569de32d758da29731687c86c635b3a3f96c201349090cf278d77c08deaea" gracePeriod=25 Nov 03 06:10:36.620503 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:36.620446 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79"] Nov 03 06:10:36.620768 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:36.620714 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" podUID="b12a4c2c-993c-448b-b051-c68fa6dc4733" containerName="maas-api" containerID="cri-o://bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198" gracePeriod=30 Nov 03 06:10:37.690128 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.690066 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:10:37.820147 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.820103 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-maas-api-tls\") pod \"b12a4c2c-993c-448b-b051-c68fa6dc4733\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " Nov 03 06:10:37.820294 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.820279 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12a4c2c-993c-448b-b051-c68fa6dc4733-kube-api-access-w9x64\" (UniqueName: \"kubernetes.io/projected/b12a4c2c-993c-448b-b051-c68fa6dc4733-kube-api-access-w9x64\") pod \"b12a4c2c-993c-448b-b051-c68fa6dc4733\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " Nov 03 06:10:37.820471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.820451 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-metrics-certs\") pod \"b12a4c2c-993c-448b-b051-c68fa6dc4733\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " Nov 03 06:10:37.820531 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.820492 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12a4c2c-993c-448b-b051-c68fa6dc4733-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b12a4c2c-993c-448b-b051-c68fa6dc4733-openshift-service-ca\") pod \"b12a4c2c-993c-448b-b051-c68fa6dc4733\" (UID: \"b12a4c2c-993c-448b-b051-c68fa6dc4733\") " Nov 03 06:10:37.820927 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.820890 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12a4c2c-993c-448b-b051-c68fa6dc4733-openshift-service-ca" pod "b12a4c2c-993c-448b-b051-c68fa6dc4733" (UID: "b12a4c2c-993c-448b-b051-c68fa6dc4733"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:10:37.821149 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.821132 4855 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b12a4c2c-993c-448b-b051-c68fa6dc4733-openshift-service-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:37.822597 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.822576 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-maas-api-tls" pod "b12a4c2c-993c-448b-b051-c68fa6dc4733" (UID: "b12a4c2c-993c-448b-b051-c68fa6dc4733"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:10:37.822877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.822861 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12a4c2c-993c-448b-b051-c68fa6dc4733-kube-api-access-w9x64" pod "b12a4c2c-993c-448b-b051-c68fa6dc4733" (UID: "b12a4c2c-993c-448b-b051-c68fa6dc4733"). InnerVolumeSpecName "kube-api-access-w9x64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:37.822935 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.822867 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-metrics-certs" pod "b12a4c2c-993c-448b-b051-c68fa6dc4733" (UID: "b12a4c2c-993c-448b-b051-c68fa6dc4733"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:10:37.923302 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.923259 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:37.923302 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.923303 4855 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b12a4c2c-993c-448b-b051-c68fa6dc4733-maas-api-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:37.923535 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.923320 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9x64\" (UniqueName: \"kubernetes.io/projected/b12a4c2c-993c-448b-b051-c68fa6dc4733-kube-api-access-w9x64\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:37.956943 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.956852 4855 generic.go:356] "Generic (PLEG): container finished" podID="b12a4c2c-993c-448b-b051-c68fa6dc4733" containerID="bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198" exitCode=0 Nov 03 06:10:37.956943 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.956904 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" Nov 03 06:10:37.957163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.956943 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" event={"ID":"b12a4c2c-993c-448b-b051-c68fa6dc4733","Type":"ContainerDied","Data":"bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198"} Nov 03 06:10:37.957163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.957002 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79" event={"ID":"b12a4c2c-993c-448b-b051-c68fa6dc4733","Type":"ContainerDied","Data":"65c0b7dd29bc0b643daffc88d14f997610f9463b8dbd151acee661b078358469"} Nov 03 06:10:37.957163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:37.957035 4855 scope.go:122] "RemoveContainer" containerID="bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198" Nov 03 06:10:38.001594 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:38.001549 4855 scope.go:122] "RemoveContainer" containerID="bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198" Nov 03 06:10:38.003637 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:38.003566 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198\": container with ID starting with bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198 not found: ID does not exist" containerID="bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198" Nov 03 06:10:38.004006 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:38.003886 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198"} err="failed to get container status \"bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198\": rpc error: code = NotFound desc = could not find container \"bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198\": container with ID starting with bff5bc08c8a3ae4a5fe231970b989b114bacb66ef57a5f0f6f4bf58dd1dd3198 not found: ID does not exist" Nov 03 06:10:38.130443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:38.092443 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79"] Nov 03 06:10:38.130443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:38.129716 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ba6892-79d54fc596-htt79"] Nov 03 06:10:38.334479 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:38.334434 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12a4c2c-993c-448b-b051-c68fa6dc4733" path="/var/lib/kubelet/pods/b12a4c2c-993c-448b-b051-c68fa6dc4733/volumes" Nov 03 06:10:39.119761 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.119726 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" containerID="cri-o://27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f" gracePeriod=45 Nov 03 06:10:39.153913 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.153880 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" containerID="cri-o://7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3" gracePeriod=45 Nov 03 06:10:39.471922 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.471896 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:10:39.475324 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.475304 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:10:39.624576 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.624542 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tmp-dir\") pod \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " Nov 03 06:10:39.625072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.624597 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-dshm\") pod \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " Nov 03 06:10:39.625072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.624637 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eea1294-b376-4523-8b28-5a213d66596c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9eea1294-b376-4523-8b28-5a213d66596c-tls-certs\") pod \"9eea1294-b376-4523-8b28-5a213d66596c\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " Nov 03 06:10:39.625072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.624690 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tls-certs\") pod \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " Nov 03 06:10:39.625072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.624727 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eea1294-b376-4523-8b28-5a213d66596c-kube-api-access-b7qzj\" (UniqueName: \"kubernetes.io/projected/9eea1294-b376-4523-8b28-5a213d66596c-kube-api-access-b7qzj\") pod \"9eea1294-b376-4523-8b28-5a213d66596c\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " Nov 03 06:10:39.625072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.624837 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-dshm\") pod \"9eea1294-b376-4523-8b28-5a213d66596c\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " Nov 03 06:10:39.625072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.624857 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-home\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-home\") pod \"9eea1294-b376-4523-8b28-5a213d66596c\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " Nov 03 06:10:39.625072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.624911 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-model-cache\") pod \"9eea1294-b376-4523-8b28-5a213d66596c\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " Nov 03 06:10:39.625072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.624944 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e84841f-e79a-4bfd-a079-cd196a8c73e9-kube-api-access-m9qzb\" (UniqueName: \"kubernetes.io/projected/5e84841f-e79a-4bfd-a079-cd196a8c73e9-kube-api-access-m9qzb\") pod \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " Nov 03 06:10:39.625072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.624964 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tmp-dir" pod "5e84841f-e79a-4bfd-a079-cd196a8c73e9" (UID: "5e84841f-e79a-4bfd-a079-cd196a8c73e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:39.625072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.625048 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-model-cache\") pod \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " Nov 03 06:10:39.625575 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.625120 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-tmp-dir\") pod \"9eea1294-b376-4523-8b28-5a213d66596c\" (UID: \"9eea1294-b376-4523-8b28-5a213d66596c\") " Nov 03 06:10:39.625575 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.625422 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-model-cache" pod "9eea1294-b376-4523-8b28-5a213d66596c" (UID: "9eea1294-b376-4523-8b28-5a213d66596c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:39.625575 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.625528 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-home\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-home\") pod \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\" (UID: \"5e84841f-e79a-4bfd-a079-cd196a8c73e9\") " Nov 03 06:10:39.626110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.626082 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-model-cache" pod "5e84841f-e79a-4bfd-a079-cd196a8c73e9" (UID: "5e84841f-e79a-4bfd-a079-cd196a8c73e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:39.626315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.626291 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-home" pod "5e84841f-e79a-4bfd-a079-cd196a8c73e9" (UID: "5e84841f-e79a-4bfd-a079-cd196a8c73e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:39.626561 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.626538 4855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-model-cache\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.626650 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.626567 4855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-model-cache\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.626650 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.626582 4855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-home\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.626650 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.626566 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-tmp-dir" pod "9eea1294-b376-4523-8b28-5a213d66596c" (UID: "9eea1294-b376-4523-8b28-5a213d66596c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:39.626650 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.626593 4855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tmp-dir\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.626859 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.626749 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-home" pod "9eea1294-b376-4523-8b28-5a213d66596c" (UID: "9eea1294-b376-4523-8b28-5a213d66596c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:39.628794 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.628737 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-dshm" pod "5e84841f-e79a-4bfd-a079-cd196a8c73e9" (UID: "5e84841f-e79a-4bfd-a079-cd196a8c73e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:39.628902 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.628817 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eea1294-b376-4523-8b28-5a213d66596c-tls-certs" pod "9eea1294-b376-4523-8b28-5a213d66596c" (UID: "9eea1294-b376-4523-8b28-5a213d66596c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:10:39.628902 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.628850 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tls-certs" pod "5e84841f-e79a-4bfd-a079-cd196a8c73e9" (UID: "5e84841f-e79a-4bfd-a079-cd196a8c73e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:10:39.629254 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.629228 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-dshm" pod "9eea1294-b376-4523-8b28-5a213d66596c" (UID: "9eea1294-b376-4523-8b28-5a213d66596c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:39.630212 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.630186 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eea1294-b376-4523-8b28-5a213d66596c-kube-api-access-b7qzj" pod "9eea1294-b376-4523-8b28-5a213d66596c" (UID: "9eea1294-b376-4523-8b28-5a213d66596c"). InnerVolumeSpecName "kube-api-access-b7qzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:39.632563 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.632533 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e84841f-e79a-4bfd-a079-cd196a8c73e9-kube-api-access-m9qzb" pod "5e84841f-e79a-4bfd-a079-cd196a8c73e9" (UID: "5e84841f-e79a-4bfd-a079-cd196a8c73e9"). InnerVolumeSpecName "kube-api-access-m9qzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:39.728232 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.728020 4855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-tmp-dir\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.728232 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.728076 4855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e84841f-e79a-4bfd-a079-cd196a8c73e9-dshm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.728232 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.728089 4855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eea1294-b376-4523-8b28-5a213d66596c-tls-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.728232 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.728103 4855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e84841f-e79a-4bfd-a079-cd196a8c73e9-tls-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.728232 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.728116 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7qzj\" (UniqueName: \"kubernetes.io/projected/9eea1294-b376-4523-8b28-5a213d66596c-kube-api-access-b7qzj\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.728232 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.728133 4855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-dshm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.728232 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.728141 4855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eea1294-b376-4523-8b28-5a213d66596c-home\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.728232 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.728153 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9qzb\" (UniqueName: \"kubernetes.io/projected/5e84841f-e79a-4bfd-a079-cd196a8c73e9-kube-api-access-m9qzb\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:39.734848 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.734818 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s"] Nov 03 06:10:39.735437 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.735416 4855 cpu_manager.go:414] "containerMap: removing container" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" Nov 03 06:10:39.752480 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752447 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" Nov 03 06:10:39.752716 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752515 4855 cpu_manager.go:414] "containerMap: removing container" podUID="b12a4c2c-993c-448b-b051-c68fa6dc4733" containerName="maas-api" Nov 03 06:10:39.752716 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752526 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12a4c2c-993c-448b-b051-c68fa6dc4733" containerName="maas-api" Nov 03 06:10:39.752716 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752650 4855 cpu_manager.go:414] "containerMap: removing container" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" Nov 03 06:10:39.752716 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752656 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" Nov 03 06:10:39.752716 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752677 4855 cpu_manager.go:414] "containerMap: removing container" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" Nov 03 06:10:39.752716 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752686 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" Nov 03 06:10:39.752716 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752714 4855 cpu_manager.go:414] "containerMap: removing container" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752741 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752783 4855 cpu_manager.go:414] "containerMap: removing container" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752790 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752805 4855 cpu_manager.go:414] "containerMap: removing container" podUID="536311c9-35e9-4309-a39a-aa592e4e3112" containerName="payload-processing" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752810 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536311c9-35e9-4309-a39a-aa592e4e3112" containerName="payload-processing" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752823 4855 cpu_manager.go:414] "containerMap: removing container" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752827 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752969 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="536311c9-35e9-4309-a39a-aa592e4e3112" containerName="payload-processing" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752980 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1bc896-f654-4773-9226-b5b099bb604b" containerName="istio-proxy" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752988 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="f485f34b-569a-4647-b63f-3b1ce1f5b2df" containerName="main" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.752996 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eea1294-b376-4523-8b28-5a213d66596c" containerName="main" Nov 03 06:10:39.753002 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.753002 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerName="main" Nov 03 06:10:39.753330 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.753009 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12a4c2c-993c-448b-b051-c68fa6dc4733" containerName="maas-api" Nov 03 06:10:39.753330 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.753016 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="74571913-1a5a-4f8a-bf23-68211e1935d3" containerName="istio-proxy" Nov 03 06:10:39.753330 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.753024 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb9581f-e4cd-4cbd-b20e-4beeff24063c" containerName="istio-proxy" Nov 03 06:10:39.756868 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.756849 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:39.763648 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.763616 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd"] Nov 03 06:10:39.776956 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.776934 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s"] Nov 03 06:10:39.777126 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.777102 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:39.797293 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.797259 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd"] Nov 03 06:10:39.844235 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.844199 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6"] Nov 03 06:10:39.849251 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.849227 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:39.878495 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.878462 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6"] Nov 03 06:10:39.909890 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.909855 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" containerID="cri-o://eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465" gracePeriod=45 Nov 03 06:10:39.930945 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.930917 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafef133-515b-4fdf-a799-cf10a42224f8-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:39.931086 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.930967 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:39.931172 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931151 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafef133-515b-4fdf-a799-cf10a42224f8-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:39.931234 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931202 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:39.931288 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931272 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:39.931339 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931309 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21990fba-a699-4ba0-8a0d-ac9025d3c129-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:39.931450 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931342 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:39.931525 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931472 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:39.931525 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931511 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:39.931604 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931590 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:39.931639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931631 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:39.931673 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931646 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:39.931673 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931668 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:39.931736 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931703 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btsk7\" (UniqueName: \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-kube-api-access-btsk7\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:39.931736 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931720 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:39.931804 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931751 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:39.931804 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931780 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:39.931860 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.931808 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wsgl\" (UniqueName: \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-kube-api-access-2wsgl\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:39.972189 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.972156 4855 generic.go:356] "Generic (PLEG): container finished" podID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" containerID="27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f" exitCode=0 Nov 03 06:10:39.972359 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.972218 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" Nov 03 06:10:39.972359 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.972269 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" event={"ID":"5e84841f-e79a-4bfd-a079-cd196a8c73e9","Type":"ContainerDied","Data":"27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f"} Nov 03 06:10:39.972359 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.972324 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj" event={"ID":"5e84841f-e79a-4bfd-a079-cd196a8c73e9","Type":"ContainerDied","Data":"a4c321f6c8f35cbd76bd025c247960552597028b7825e5e6ce36f076df093bbf"} Nov 03 06:10:39.972359 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.972349 4855 scope.go:122] "RemoveContainer" containerID="27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f" Nov 03 06:10:39.973937 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.973915 4855 generic.go:356] "Generic (PLEG): container finished" podID="9eea1294-b376-4523-8b28-5a213d66596c" containerID="7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3" exitCode=0 Nov 03 06:10:39.974033 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.973944 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" event={"ID":"9eea1294-b376-4523-8b28-5a213d66596c","Type":"ContainerDied","Data":"7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3"} Nov 03 06:10:39.974033 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.973963 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" Nov 03 06:10:39.974033 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:39.973983 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2" event={"ID":"9eea1294-b376-4523-8b28-5a213d66596c","Type":"ContainerDied","Data":"ac40ec62af0329e9cf9bd2c7c1d6626d9514f8f19f130c29a7d145319685dac3"} Nov 03 06:10:40.022438 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.022378 4855 scope.go:122] "RemoveContainer" containerID="27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f" Nov 03 06:10:40.022823 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:40.022799 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f\": container with ID starting with 27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f not found: ID does not exist" containerID="27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f" Nov 03 06:10:40.022888 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.022834 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f"} err="failed to get container status \"27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f\": rpc error: code = NotFound desc = could not find container \"27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f\": container with ID starting with 27a231c68fb3dd935d6828dc45167132aa58721f9cbce1b99ddd66073d15760f not found: ID does not exist" Nov 03 06:10:40.022888 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.022854 4855 scope.go:122] "RemoveContainer" containerID="7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3" Nov 03 06:10:40.032429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.032383 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2"] Nov 03 06:10:40.032759 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.032656 4855 scope.go:122] "RemoveContainer" containerID="7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3" Nov 03 06:10:40.033103 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:40.033039 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3\": container with ID starting with 7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3 not found: ID does not exist" containerID="7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3" Nov 03 06:10:40.033103 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033079 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3"} err="failed to get container status \"7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3\": rpc error: code = NotFound desc = could not find container \"7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3\": container with ID starting with 7a4e29cca2a65b114561da365fa017ed0b545227e523c002861f6450ee5725d3 not found: ID does not exist" Nov 03 06:10:40.033442 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033421 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafef133-515b-4fdf-a799-cf10a42224f8-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.033514 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033482 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.033580 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033522 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.033580 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033558 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.033728 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033706 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.033785 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033762 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.033857 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033835 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafef133-515b-4fdf-a799-cf10a42224f8-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.033905 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033889 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.033951 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033935 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.034003 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033965 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.034003 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.033995 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21990fba-a699-4ba0-8a0d-ac9025d3c129-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.034110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034021 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.034110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034069 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.034110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034099 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.034282 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034140 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.034282 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034124 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafef133-515b-4fdf-a799-cf10a42224f8-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.034282 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034228 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.034877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034577 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.034877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034658 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.034877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034684 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.034877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034719 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.034877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034726 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.034877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034756 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.034877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034778 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.034877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034836 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.035321 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034842 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.035321 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034921 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9pz\" (UniqueName: \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-kube-api-access-gq9pz\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.035321 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034965 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.035321 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.034982 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btsk7\" (UniqueName: \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-kube-api-access-btsk7\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.035321 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.035012 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.035321 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.035029 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.035321 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.035128 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21990fba-a699-4ba0-8a0d-ac9025d3c129-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.035321 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.035142 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.035321 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.035196 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.035834 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.035498 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.035834 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.035239 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wsgl\" (UniqueName: \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-kube-api-access-2wsgl\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.035834 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.035613 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.037253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.037229 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.037338 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.037229 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.037694 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.037666 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafef133-515b-4fdf-a799-cf10a42224f8-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.037781 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.037772 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.045212 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.045192 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/simulator-w4-be1a66-kserve-7fdd59dd46-928z2"] Nov 03 06:10:40.071057 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.071018 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.071461 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.071436 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wsgl\" (UniqueName: \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-kube-api-access-2wsgl\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.071623 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.071602 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.073183 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.073157 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btsk7\" (UniqueName: \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-kube-api-access-btsk7\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.078352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.078326 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:40.099530 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.099485 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj"] Nov 03 06:10:40.099826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.099749 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:40.100570 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.100534 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-2-w4-be1a66-kserve-59946ff55c-w9ccj"] Nov 03 06:10:40.136889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.136848 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.138899 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.138875 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.139060 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.139040 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9pz\" (UniqueName: \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-kube-api-access-gq9pz\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.139306 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.139288 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.139524 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.139501 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.139651 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.139636 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.139771 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.139759 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.139882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.139869 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.140020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.140008 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.140603 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.140577 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.141003 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.140981 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.142797 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.142770 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.170686 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.169219 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.172557 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.172527 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9pz\" (UniqueName: \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-kube-api-access-gq9pz\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.178223 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.177194 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.178223 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.177709 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.178223 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.177852 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.210858 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.207341 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.288957 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.288710 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:10:40.347681 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.347644 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e84841f-e79a-4bfd-a079-cd196a8c73e9" path="/var/lib/kubelet/pods/5e84841f-e79a-4bfd-a079-cd196a8c73e9/volumes" Nov 03 06:10:40.348198 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.348177 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eea1294-b376-4523-8b28-5a213d66596c" path="/var/lib/kubelet/pods/9eea1294-b376-4523-8b28-5a213d66596c/volumes" Nov 03 06:10:40.445566 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.445531 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-tmp-dir\") pod \"b7f0b7ce-07b2-43be-819f-cde40d902902\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " Nov 03 06:10:40.445738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.445659 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f0b7ce-07b2-43be-819f-cde40d902902-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f0b7ce-07b2-43be-819f-cde40d902902-tls-certs\") pod \"b7f0b7ce-07b2-43be-819f-cde40d902902\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " Nov 03 06:10:40.445738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.445726 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-model-cache\") pod \"b7f0b7ce-07b2-43be-819f-cde40d902902\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " Nov 03 06:10:40.445858 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.445793 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f0b7ce-07b2-43be-819f-cde40d902902-kube-api-access-ph7s9\" (UniqueName: \"kubernetes.io/projected/b7f0b7ce-07b2-43be-819f-cde40d902902-kube-api-access-ph7s9\") pod \"b7f0b7ce-07b2-43be-819f-cde40d902902\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " Nov 03 06:10:40.445858 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.445836 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-dshm\") pod \"b7f0b7ce-07b2-43be-819f-cde40d902902\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " Nov 03 06:10:40.445958 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.445862 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-home\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-home\") pod \"b7f0b7ce-07b2-43be-819f-cde40d902902\" (UID: \"b7f0b7ce-07b2-43be-819f-cde40d902902\") " Nov 03 06:10:40.446527 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.446260 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-model-cache" pod "b7f0b7ce-07b2-43be-819f-cde40d902902" (UID: "b7f0b7ce-07b2-43be-819f-cde40d902902"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:40.446527 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.446460 4855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-model-cache\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:40.447267 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.446936 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-tmp-dir" pod "b7f0b7ce-07b2-43be-819f-cde40d902902" (UID: "b7f0b7ce-07b2-43be-819f-cde40d902902"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:40.447842 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.447811 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-home" pod "b7f0b7ce-07b2-43be-819f-cde40d902902" (UID: "b7f0b7ce-07b2-43be-819f-cde40d902902"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:40.449907 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.449874 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-dshm" pod "b7f0b7ce-07b2-43be-819f-cde40d902902" (UID: "b7f0b7ce-07b2-43be-819f-cde40d902902"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:40.449985 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.449899 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f0b7ce-07b2-43be-819f-cde40d902902-tls-certs" pod "b7f0b7ce-07b2-43be-819f-cde40d902902" (UID: "b7f0b7ce-07b2-43be-819f-cde40d902902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:10:40.451723 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.451698 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f0b7ce-07b2-43be-819f-cde40d902902-kube-api-access-ph7s9" pod "b7f0b7ce-07b2-43be-819f-cde40d902902" (UID: "b7f0b7ce-07b2-43be-819f-cde40d902902"). InnerVolumeSpecName "kube-api-access-ph7s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:40.465096 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.464913 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:40.487289 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.487054 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s"] Nov 03 06:10:40.490052 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.490018 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:40.490148 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.490096 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:40.490148 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.490135 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:40.491696 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.491675 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd"] Nov 03 06:10:40.495252 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:10:40.495142 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafef133_515b_4fdf_a799_cf10a42224f8.slice/crio-b1fd3d7aa981b4b3dd068c5a4d2f51774b9c91377a00e0e0206e6ee18f0979cd WatchSource:0}: Error finding container b1fd3d7aa981b4b3dd068c5a4d2f51774b9c91377a00e0e0206e6ee18f0979cd: Status 404 returned error can't find the container with id b1fd3d7aa981b4b3dd068c5a4d2f51774b9c91377a00e0e0206e6ee18f0979cd Nov 03 06:10:40.503648 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.503611 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:40.503729 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.503694 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:40.503773 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.503739 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:40.554008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.553889 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7s9\" (UniqueName: \"kubernetes.io/projected/b7f0b7ce-07b2-43be-819f-cde40d902902-kube-api-access-ph7s9\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:40.554008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.553922 4855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-dshm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:40.554008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.553938 4855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-home\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:40.554008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.553950 4855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f0b7ce-07b2-43be-819f-cde40d902902-tmp-dir\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:40.554008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.553962 4855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f0b7ce-07b2-43be-819f-cde40d902902-tls-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:40.914864 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.913931 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6"] Nov 03 06:10:40.917441 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.915188 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:40.917441 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.915280 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:40.917441 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.915324 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:10:40.985914 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.985881 4855 generic.go:356] "Generic (PLEG): container finished" podID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerID="eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465" exitCode=0 Nov 03 06:10:40.986065 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.985935 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" Nov 03 06:10:40.986065 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.985956 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" event={"ID":"b7f0b7ce-07b2-43be-819f-cde40d902902","Type":"ContainerDied","Data":"eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465"} Nov 03 06:10:40.986065 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.986006 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8" event={"ID":"b7f0b7ce-07b2-43be-819f-cde40d902902","Type":"ContainerDied","Data":"3116c1277a9d7c693bbc1e8c97bf2af78ddb7af3b8cbb067593ddf6e266bc272"} Nov 03 06:10:40.986065 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.986032 4855 scope.go:122] "RemoveContainer" containerID="eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465" Nov 03 06:10:40.996651 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.994770 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" event={"ID":"cafef133-515b-4fdf-a799-cf10a42224f8","Type":"ContainerStarted","Data":"e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8"} Nov 03 06:10:40.996651 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.994829 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" event={"ID":"cafef133-515b-4fdf-a799-cf10a42224f8","Type":"ContainerStarted","Data":"b1fd3d7aa981b4b3dd068c5a4d2f51774b9c91377a00e0e0206e6ee18f0979cd"} Nov 03 06:10:40.999474 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.999450 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" event={"ID":"21990fba-a699-4ba0-8a0d-ac9025d3c129","Type":"ContainerStarted","Data":"fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8"} Nov 03 06:10:40.999981 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:40.999941 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" event={"ID":"21990fba-a699-4ba0-8a0d-ac9025d3c129","Type":"ContainerStarted","Data":"ec3762b4385877197c61e1fe4a475ac2199dcb7f33ab188674bad7cc35029c86"} Nov 03 06:10:41.001778 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.001738 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" event={"ID":"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62","Type":"ContainerStarted","Data":"9ed8fb493e9ed3f246fc92483dcc32c2cf909d259a3eb5a8037309af3eeaaa04"} Nov 03 06:10:41.002048 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.002028 4855 scope.go:122] "RemoveContainer" containerID="eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465" Nov 03 06:10:41.002509 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:41.002483 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465\": container with ID starting with eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465 not found: ID does not exist" containerID="eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465" Nov 03 06:10:41.002602 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.002519 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465"} err="failed to get container status \"eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465\": rpc error: code = NotFound desc = could not find container \"eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465\": container with ID starting with eab4a9b6f5c47a35924908cab30931e807024b54ba0b1428d99a120a8a1ef465 not found: ID does not exist" Nov 03 06:10:41.040326 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.040297 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn"] Nov 03 06:10:41.070910 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.070840 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podStartSLOduration=2.070818015 podStartE2EDuration="2.070818015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:41.063785643 +0000 UTC m=+928.872172911" watchObservedRunningTime="2026-11-03 06:10:41.070818015 +0000 UTC m=+928.879205278" Nov 03 06:10:41.079309 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.079180 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:41.080297 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.080272 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 06:10:41.080407 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.080320 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 06:10:41.102633 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.102567 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:41.102981 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.102958 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 06:10:41.103192 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.103170 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 06:10:41.111268 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.111241 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8"] Nov 03 06:10:41.116806 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.116783 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-86728b/distinct-w6-86728b-kserve-6894c57c46-pg7t8"] Nov 03 06:10:41.193571 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:41.193510 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podStartSLOduration=2.1934941119999998 podStartE2EDuration="2.193494112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:41.180548086 +0000 UTC m=+928.988935357" watchObservedRunningTime="2026-11-03 06:10:41.193494112 +0000 UTC m=+929.001881380" Nov 03 06:10:42.011567 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:42.011523 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" event={"ID":"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62","Type":"ContainerStarted","Data":"9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798"} Nov 03 06:10:42.085410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:42.084824 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 06:10:42.085410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:42.084890 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 06:10:42.085838 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:42.085786 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podStartSLOduration=3.085769382 podStartE2EDuration="3.085769382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:42.080913654 +0000 UTC m=+929.889300923" watchObservedRunningTime="2026-11-03 06:10:42.085769382 +0000 UTC m=+929.894156650" Nov 03 06:10:42.100325 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:42.100287 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 06:10:42.100588 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:42.100561 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 06:10:42.425202 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:42.425154 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" path="/var/lib/kubelet/pods/b7f0b7ce-07b2-43be-819f-cde40d902902/volumes" Nov 03 06:10:42.468413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:42.467468 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:42.468413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:42.467625 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 06:10:42.468413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:42.467703 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 06:10:43.088963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:43.088928 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 06:10:43.089298 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:43.089251 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 06:10:43.105075 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:43.105040 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 06:10:43.106756 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:43.106724 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 06:10:43.475957 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:43.475859 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 06:10:43.477019 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:43.476979 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 06:10:43.697420 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:43.696508 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" podUID="97ba4903-d54a-4291-bd8b-811bc6b3302a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 06:10:44.116422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:44.109054 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 06:10:44.116422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:44.109122 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 06:10:44.116422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:44.113529 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 06:10:44.116422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:44.113582 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 06:10:44.483469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:44.477001 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 06:10:44.483469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:44.477061 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 06:10:45.093413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:45.081816 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 06:10:45.093413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:45.081878 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 06:10:45.112543 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:45.112497 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 06:10:45.112804 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:45.112777 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 06:10:45.476468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:45.468789 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 06:10:45.476468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:45.468851 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 06:10:46.131411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.123824 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 06:10:46.131411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.123931 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 06:10:46.131411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.126003 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 06:10:46.131411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.126057 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 06:10:46.214488 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.212473 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm"] Nov 03 06:10:46.214488 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.212936 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerName="istio-proxy" containerID="cri-o://ed53830283109f24d4dc6403b86c5ddb661aebd8d517b705099bd6410be4b49f" gracePeriod=30 Nov 03 06:10:46.482733 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.482632 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:10:46.483527 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.483486 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 06:10:46.483729 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.483707 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 06:10:46.683415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.667382 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z"] Nov 03 06:10:46.683751 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.683725 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b"] Nov 03 06:10:46.684627 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.684053 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" containerID="cri-o://1f1c0a835df895e23dedc8a235501fedda744f1ce2a51b0ece23e1a0005d5271" gracePeriod=30 Nov 03 06:10:46.713639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:46.713605 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" containerID="cri-o://f76325e9988ab6f7b56ba91e4de6be40b61567e73c751ef405c6621f27442604" gracePeriod=30 Nov 03 06:10:47.077085 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:47.077048 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:10:47.077373 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:47.077347 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:10:47.088721 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:47.088686 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 06:10:47.088981 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:47.088951 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 06:10:47.101961 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:47.101921 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 06:10:47.102247 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:47.102221 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 06:10:47.485470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:47.481955 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 06:10:47.485470 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:47.482021 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 06:10:48.081615 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:48.081577 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 06:10:48.081902 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:48.081874 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 06:10:48.104280 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:48.104242 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 06:10:48.104590 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:48.104561 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 06:10:48.467474 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:48.466705 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 06:10:48.467474 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:48.466776 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 06:10:49.048483 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:49.048434 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 06:10:49.048903 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:49.048507 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:10:49.079838 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:49.079793 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 06:10:49.080034 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:49.079866 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 06:10:49.100340 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:49.100299 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 06:10:49.100575 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:49.100372 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 06:10:49.466899 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:49.466803 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 06:10:49.467230 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:49.467196 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 06:10:49.894653 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:49.894615 4855 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:10:49.894913 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:49.894882 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:10:50.084085 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:50.084053 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:50.084554 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:50.084480 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:50.085371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:50.085351 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:10:50.134970 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:50.134938 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:50.135718 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:50.135691 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:50.136906 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:50.136871 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:10:50.293828 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:50.293738 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:10:50.470418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:50.470363 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:50.470704 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:50.470674 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:50.471428 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:50.471410 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:10:51.912494 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:51.897838 4855 generic.go:356] "Generic (PLEG): container finished" podID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerID="ed53830283109f24d4dc6403b86c5ddb661aebd8d517b705099bd6410be4b49f" exitCode=0 Nov 03 06:10:51.912494 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:51.898943 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" event={"ID":"c4fbec74-f9e1-4d46-abcc-cedad8e517fd","Type":"ContainerDied","Data":"ed53830283109f24d4dc6403b86c5ddb661aebd8d517b705099bd6410be4b49f"} Nov 03 06:10:52.117495 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:52.116025 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:53.023415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.023322 4855 generic.go:356] "Generic (PLEG): container finished" podID="299590cc-de3d-4408-8653-320887002525" containerID="f76325e9988ab6f7b56ba91e4de6be40b61567e73c751ef405c6621f27442604" exitCode=0 Nov 03 06:10:53.023947 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.023456 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" event={"ID":"299590cc-de3d-4408-8653-320887002525","Type":"ContainerDied","Data":"f76325e9988ab6f7b56ba91e4de6be40b61567e73c751ef405c6621f27442604"} Nov 03 06:10:53.023947 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.023529 4855 scope.go:122] "RemoveContainer" containerID="6bafdf370835679a5d1fa10ac864ba0fef9f5d1933cec2f0e1c95b6c7d86e03b" Nov 03 06:10:53.052416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.042564 4855 generic.go:356] "Generic (PLEG): container finished" podID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerID="1f1c0a835df895e23dedc8a235501fedda744f1ce2a51b0ece23e1a0005d5271" exitCode=0 Nov 03 06:10:53.052416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.042785 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" event={"ID":"0e2e8f6c-d325-4e8c-9f49-93477681daab","Type":"ContainerDied","Data":"1f1c0a835df895e23dedc8a235501fedda744f1ce2a51b0ece23e1a0005d5271"} Nov 03 06:10:53.123416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.122661 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:10:53.123416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.123342 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:10:53.449688 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.449639 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:53.470418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.451190 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded" Nov 03 06:10:53.524005 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.487053 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:10:53.634533 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.634481 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:53.786762 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:53.786722 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" podUID="97ba4903-d54a-4291-bd8b-811bc6b3302a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 06:10:54.090419 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:54.064316 4855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:54.134366 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:54.134083 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 06:10:54.943339 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:54.921621 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:54.973709 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:54.973638 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:10:54.991454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:54.987467 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:55.150459 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.125294 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:10:55.168217 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.168171 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:10:55.171342 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.171321 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:10:55.172359 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.155667 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:10:55.261512 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.244340 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:55.836840 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.824811 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-data\") pod \"299590cc-de3d-4408-8653-320887002525\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " Nov 03 06:10:55.905506 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.905472 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-kube-api-access-g6bql\" (UniqueName: \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-kube-api-access-g6bql\") pod \"299590cc-de3d-4408-8653-320887002525\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " Nov 03 06:10:55.930563 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.930525 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-istio-token\" (UniqueName: \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-istio-token\") pod \"299590cc-de3d-4408-8653-320887002525\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " Nov 03 06:10:55.930989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.930968 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299590cc-de3d-4408-8653-320887002525-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299590cc-de3d-4408-8653-320887002525-istiod-ca-cert\") pod \"299590cc-de3d-4408-8653-320887002525\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " Nov 03 06:10:55.932928 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.932909 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-envoy\") pod \"299590cc-de3d-4408-8653-320887002525\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " Nov 03 06:10:55.937869 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.937847 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-credential-socket\") pod \"299590cc-de3d-4408-8653-320887002525\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " Nov 03 06:10:55.941801 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.941778 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-certs\") pod \"299590cc-de3d-4408-8653-320887002525\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " Nov 03 06:10:55.942136 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.942117 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-socket\") pod \"299590cc-de3d-4408-8653-320887002525\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " Nov 03 06:10:55.942378 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.942359 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/299590cc-de3d-4408-8653-320887002525-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299590cc-de3d-4408-8653-320887002525-istio-podinfo\") pod \"299590cc-de3d-4408-8653-320887002525\" (UID: \"299590cc-de3d-4408-8653-320887002525\") " Nov 03 06:10:55.945894 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.837100 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-data" pod "299590cc-de3d-4408-8653-320887002525" (UID: "299590cc-de3d-4408-8653-320887002525"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:55.946059 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.931699 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299590cc-de3d-4408-8653-320887002525-istiod-ca-cert" pod "299590cc-de3d-4408-8653-320887002525" (UID: "299590cc-de3d-4408-8653-320887002525"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:10:55.969446 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.949551 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-credential-socket" pod "299590cc-de3d-4408-8653-320887002525" (UID: "299590cc-de3d-4408-8653-320887002525"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:55.969446 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.950915 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299590cc-de3d-4408-8653-320887002525-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:55.969446 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.950940 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:55.969446 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.950955 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:55.989415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.930360 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:10:55.989415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.988497 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:55.989676 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:55.989518 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-certs" pod "299590cc-de3d-4408-8653-320887002525" (UID: "299590cc-de3d-4408-8653-320887002525"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:56.013065 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.009824 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-socket" pod "299590cc-de3d-4408-8653-320887002525" (UID: "299590cc-de3d-4408-8653-320887002525"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:56.048416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.029023 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:10:56.090480 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.090431 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:56.851093 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.851067 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:56.876264 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.213466 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:10:56.878888 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.215959 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:10:56.894859 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.240571 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" containerID="cri-o://98020769a3dbbce0f9c7eac3f06d68cc0aaffac8400e053c094707f2ff860dab" gracePeriod=45 Nov 03 06:10:56.896248 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.252301 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:56.896802 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.429855 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" Nov 03 06:10:56.897190 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.850990 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" event={"ID":"c4fbec74-f9e1-4d46-abcc-cedad8e517fd","Type":"ContainerDied","Data":"88484b362aa5b11c2299fa8ee76e72bd22c83ab7a61ed83dbbdc280ca1b305df"} Nov 03 06:10:56.897436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.897411 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b" event={"ID":"299590cc-de3d-4408-8653-320887002525","Type":"ContainerDied","Data":"cacb752163b30657e9da9ad2413c2a43b2d57539d0e763e687098f2fd229032d"} Nov 03 06:10:56.897621 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.897601 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" event={"ID":"0e2e8f6c-d325-4e8c-9f49-93477681daab","Type":"ContainerDied","Data":"2ac1ffc4de6cd395eec77d64a74fbebb1f5b46229d3bebebcad2810399909abf"} Nov 03 06:10:56.897808 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.897777 4855 scope.go:122] "RemoveContainer" containerID="ed53830283109f24d4dc6403b86c5ddb661aebd8d517b705099bd6410be4b49f" Nov 03 06:10:56.898061 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.143258 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-envoy" pod "299590cc-de3d-4408-8653-320887002525" (UID: "299590cc-de3d-4408-8653-320887002525"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:56.898221 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.153699 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/299590cc-de3d-4408-8653-320887002525-istio-podinfo" pod "299590cc-de3d-4408-8653-320887002525" (UID: "299590cc-de3d-4408-8653-320887002525"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:10:56.898338 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.257305 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-kube-api-access-g6bql" pod "299590cc-de3d-4408-8653-320887002525" (UID: "299590cc-de3d-4408-8653-320887002525"). InnerVolumeSpecName "kube-api-access-g6bql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:56.898511 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.363613 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-istio-token" pod "299590cc-de3d-4408-8653-320887002525" (UID: "299590cc-de3d-4408-8653-320887002525"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:56.977343 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.977312 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-envoy\") pod \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " Nov 03 06:10:56.977725 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.977691 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-kube-api-access-xdrkh\" (UniqueName: \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-kube-api-access-xdrkh\") pod \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " Nov 03 06:10:56.977940 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.977912 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-socket\") pod \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " Nov 03 06:10:56.978074 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.978059 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-envoy\") pod \"0e2e8f6c-d325-4e8c-9f49-93477681daab\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " Nov 03 06:10:56.978211 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.978197 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-token\" (UniqueName: \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-token\") pod \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " Nov 03 06:10:56.978338 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.978322 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-credential-socket\") pod \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " Nov 03 06:10:56.978665 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.978646 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-certs\") pod \"0e2e8f6c-d325-4e8c-9f49-93477681daab\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " Nov 03 06:10:56.978802 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.978787 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-certs\") pod \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " Nov 03 06:10:56.978922 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.978907 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e2e8f6c-d325-4e8c-9f49-93477681daab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e2e8f6c-d325-4e8c-9f49-93477681daab-istiod-ca-cert\") pod \"0e2e8f6c-d325-4e8c-9f49-93477681daab\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " Nov 03 06:10:56.979043 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.979028 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-kube-api-access-vc965\" (UniqueName: \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-kube-api-access-vc965\") pod \"0e2e8f6c-d325-4e8c-9f49-93477681daab\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " Nov 03 06:10:56.979168 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.979152 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-data\") pod \"0e2e8f6c-d325-4e8c-9f49-93477681daab\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " Nov 03 06:10:56.979325 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.979310 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-data\") pod \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " Nov 03 06:10:56.979464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.979447 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-podinfo\") pod \"0e2e8f6c-d325-4e8c-9f49-93477681daab\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " Nov 03 06:10:56.979584 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.979568 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istiod-ca-cert\") pod \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " Nov 03 06:10:56.979715 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.979700 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-podinfo\") pod \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\" (UID: \"c4fbec74-f9e1-4d46-abcc-cedad8e517fd\") " Nov 03 06:10:56.979871 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.979856 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-credential-socket\") pod \"0e2e8f6c-d325-4e8c-9f49-93477681daab\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " Nov 03 06:10:56.979998 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.979982 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-socket\") pod \"0e2e8f6c-d325-4e8c-9f49-93477681daab\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " Nov 03 06:10:56.980172 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.980156 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-token\" (UniqueName: \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-token\") pod \"0e2e8f6c-d325-4e8c-9f49-93477681daab\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " Nov 03 06:10:56.981105 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.981082 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299590cc-de3d-4408-8653-320887002525-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:56.981293 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.981279 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6bql\" (UniqueName: \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-kube-api-access-g6bql\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:56.981434 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.981382 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299590cc-de3d-4408-8653-320887002525-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:56.982148 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.982133 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299590cc-de3d-4408-8653-320887002525-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:56.982260 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.981809 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-certs" pod "0e2e8f6c-d325-4e8c-9f49-93477681daab" (UID: "0e2e8f6c-d325-4e8c-9f49-93477681daab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:56.982351 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.982324 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2e8f6c-d325-4e8c-9f49-93477681daab-istiod-ca-cert" pod "0e2e8f6c-d325-4e8c-9f49-93477681daab" (UID: "0e2e8f6c-d325-4e8c-9f49-93477681daab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:10:56.982439 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.982019 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-certs" pod "c4fbec74-f9e1-4d46-abcc-cedad8e517fd" (UID: "c4fbec74-f9e1-4d46-abcc-cedad8e517fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:56.982736 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.982709 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istiod-ca-cert" pod "c4fbec74-f9e1-4d46-abcc-cedad8e517fd" (UID: "c4fbec74-f9e1-4d46-abcc-cedad8e517fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:10:56.982984 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.982961 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-data" pod "0e2e8f6c-d325-4e8c-9f49-93477681daab" (UID: "0e2e8f6c-d325-4e8c-9f49-93477681daab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:56.983170 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.983148 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-data" pod "c4fbec74-f9e1-4d46-abcc-cedad8e517fd" (UID: "c4fbec74-f9e1-4d46-abcc-cedad8e517fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:56.983731 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.983711 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-socket" pod "c4fbec74-f9e1-4d46-abcc-cedad8e517fd" (UID: "c4fbec74-f9e1-4d46-abcc-cedad8e517fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:56.983854 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.983793 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-credential-socket" pod "c4fbec74-f9e1-4d46-abcc-cedad8e517fd" (UID: "c4fbec74-f9e1-4d46-abcc-cedad8e517fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:56.994825 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.994782 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 06:10:56.994955 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.994782 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:10:56.999656 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:56.999344 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-credential-socket" pod "0e2e8f6c-d325-4e8c-9f49-93477681daab" (UID: "0e2e8f6c-d325-4e8c-9f49-93477681daab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:57.003564 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.003466 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-socket" pod "0e2e8f6c-d325-4e8c-9f49-93477681daab" (UID: "0e2e8f6c-d325-4e8c-9f49-93477681daab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:57.074414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.069267 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:10:57.074414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.073098 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-podinfo" pod "c4fbec74-f9e1-4d46-abcc-cedad8e517fd" (UID: "c4fbec74-f9e1-4d46-abcc-cedad8e517fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:10:57.074414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.073742 4855 scope.go:122] "RemoveContainer" containerID="f76325e9988ab6f7b56ba91e4de6be40b61567e73c751ef405c6621f27442604" Nov 03 06:10:57.109569 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.109517 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-podinfo" pod "0e2e8f6c-d325-4e8c-9f49-93477681daab" (UID: "0e2e8f6c-d325-4e8c-9f49-93477681daab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:10:57.109947 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.109927 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-podinfo\") pod \"0e2e8f6c-d325-4e8c-9f49-93477681daab\" (UID: \"0e2e8f6c-d325-4e8c-9f49-93477681daab\") " Nov 03 06:10:57.111795 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.111760 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.111974 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.111958 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.112089 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.112074 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.112249 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.112237 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.112490 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.112477 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e2e8f6c-d325-4e8c-9f49-93477681daab-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.112638 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.112625 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.112745 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.112734 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.112874 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.112864 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.112977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.112967 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.113149 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.113134 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.120049 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.120000 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.120437 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:10:57.112839 4855 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/0e2e8f6c-d325-4e8c-9f49-93477681daab/volumes/kubernetes.io~downward-api/istio-podinfo Nov 03 06:10:57.121641 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.121618 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-podinfo" pod "0e2e8f6c-d325-4e8c-9f49-93477681daab" (UID: "0e2e8f6c-d325-4e8c-9f49-93477681daab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:10:57.150074 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.143200 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-token" pod "0e2e8f6c-d325-4e8c-9f49-93477681daab" (UID: "0e2e8f6c-d325-4e8c-9f49-93477681daab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:57.163051 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.163021 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" Nov 03 06:10:57.164208 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.164184 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b"] Nov 03 06:10:57.170421 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.169380 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-vvr6b"] Nov 03 06:10:57.183439 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.181686 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-kube-api-access-xdrkh" pod "c4fbec74-f9e1-4d46-abcc-cedad8e517fd" (UID: "c4fbec74-f9e1-4d46-abcc-cedad8e517fd"). InnerVolumeSpecName "kube-api-access-xdrkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:57.183439 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.182201 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": EOF" Nov 03 06:10:57.183771 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.183746 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-envoy" pod "0e2e8f6c-d325-4e8c-9f49-93477681daab" (UID: "0e2e8f6c-d325-4e8c-9f49-93477681daab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:57.195405 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.188622 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-envoy" pod "c4fbec74-f9e1-4d46-abcc-cedad8e517fd" (UID: "c4fbec74-f9e1-4d46-abcc-cedad8e517fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:57.201752 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.200959 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:10:57.250989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.222561 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-token" pod "c4fbec74-f9e1-4d46-abcc-cedad8e517fd" (UID: "c4fbec74-f9e1-4d46-abcc-cedad8e517fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:57.250989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.223808 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.250989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.223833 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.250989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.223846 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.250989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.223859 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdrkh\" (UniqueName: \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-kube-api-access-xdrkh\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.250989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.223874 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e2e8f6c-d325-4e8c-9f49-93477681daab-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.250989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.223887 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fbec74-f9e1-4d46-abcc-cedad8e517fd-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.250989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.248926 4855 scope.go:122] "RemoveContainer" containerID="1f1c0a835df895e23dedc8a235501fedda744f1ce2a51b0ece23e1a0005d5271" Nov 03 06:10:57.250989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.249220 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:10:57.296912 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.293102 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-kube-api-access-vc965" pod "0e2e8f6c-d325-4e8c-9f49-93477681daab" (UID: "0e2e8f6c-d325-4e8c-9f49-93477681daab"). InnerVolumeSpecName "kube-api-access-vc965". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:57.326852 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.326475 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc965\" (UniqueName: \"kubernetes.io/projected/0e2e8f6c-d325-4e8c-9f49-93477681daab-kube-api-access-vc965\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.404205 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.404155 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": dial tcp 10.133.0.57:8000: connect: connection refused" Nov 03 06:10:57.468570 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.468524 4855 generic.go:356] "Generic (PLEG): container finished" podID="97ba4903-d54a-4291-bd8b-811bc6b3302a" containerID="e4f569de32d758da29731687c86c635b3a3f96c201349090cf278d77c08deaea" exitCode=0 Nov 03 06:10:57.468717 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.468626 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" event={"ID":"97ba4903-d54a-4291-bd8b-811bc6b3302a","Type":"ContainerDied","Data":"e4f569de32d758da29731687c86c635b3a3f96c201349090cf278d77c08deaea"} Nov 03 06:10:57.473611 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.473583 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z" Nov 03 06:10:57.475877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.475847 4855 generic.go:356] "Generic (PLEG): container finished" podID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerID="98020769a3dbbce0f9c7eac3f06d68cc0aaffac8400e053c094707f2ff860dab" exitCode=0 Nov 03 06:10:57.476041 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.476021 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" event={"ID":"f80d1e19-3ce6-4060-8305-7cfec3f4c296","Type":"ContainerDied","Data":"98020769a3dbbce0f9c7eac3f06d68cc0aaffac8400e053c094707f2ff860dab"} Nov 03 06:10:57.477142 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.477123 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm" Nov 03 06:10:57.479034 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.479012 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:10:57.518752 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.518681 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:10:57.528086 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.528053 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ba4903-d54a-4291-bd8b-811bc6b3302a-kube-api-access-82rtl\" (UniqueName: \"kubernetes.io/projected/97ba4903-d54a-4291-bd8b-811bc6b3302a-kube-api-access-82rtl\") pod \"97ba4903-d54a-4291-bd8b-811bc6b3302a\" (UID: \"97ba4903-d54a-4291-bd8b-811bc6b3302a\") " Nov 03 06:10:57.528249 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.528224 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ba4903-d54a-4291-bd8b-811bc6b3302a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ba4903-d54a-4291-bd8b-811bc6b3302a-plugins-config-volume\") pod \"97ba4903-d54a-4291-bd8b-811bc6b3302a\" (UID: \"97ba4903-d54a-4291-bd8b-811bc6b3302a\") " Nov 03 06:10:57.528685 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.528658 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ba4903-d54a-4291-bd8b-811bc6b3302a-plugins-config-volume" pod "97ba4903-d54a-4291-bd8b-811bc6b3302a" (UID: "97ba4903-d54a-4291-bd8b-811bc6b3302a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:10:57.528818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.528799 4855 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ba4903-d54a-4291-bd8b-811bc6b3302a-plugins-config-volume\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.530253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.530233 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ba4903-d54a-4291-bd8b-811bc6b3302a-kube-api-access-82rtl" pod "97ba4903-d54a-4291-bd8b-811bc6b3302a" (UID: "97ba4903-d54a-4291-bd8b-811bc6b3302a"). InnerVolumeSpecName "kube-api-access-82rtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:57.531429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.531412 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm"] Nov 03 06:10:57.538116 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.538090 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-2gnrm"] Nov 03 06:10:57.578219 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.578196 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z"] Nov 03 06:10:57.580788 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.580764 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-87p7z"] Nov 03 06:10:57.630163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.630132 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80d1e19-3ce6-4060-8305-7cfec3f4c296-kube-api-access-2c5f4\" (UniqueName: \"kubernetes.io/projected/f80d1e19-3ce6-4060-8305-7cfec3f4c296-kube-api-access-2c5f4\") pod \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " Nov 03 06:10:57.630315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.630227 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-home\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-home\") pod \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " Nov 03 06:10:57.630315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.630279 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-dshm\") pod \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " Nov 03 06:10:57.630497 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.630442 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-model-cache\") pod \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " Nov 03 06:10:57.630566 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.630509 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-home" pod "f80d1e19-3ce6-4060-8305-7cfec3f4c296" (UID: "f80d1e19-3ce6-4060-8305-7cfec3f4c296"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:57.630566 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.630518 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tmp-dir\") pod \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " Nov 03 06:10:57.630672 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.630645 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tls-certs\" (UniqueName: \"kubernetes.io/secret/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tls-certs\") pod \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\" (UID: \"f80d1e19-3ce6-4060-8305-7cfec3f4c296\") " Nov 03 06:10:57.630756 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.630719 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tmp-dir" pod "f80d1e19-3ce6-4060-8305-7cfec3f4c296" (UID: "f80d1e19-3ce6-4060-8305-7cfec3f4c296"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:57.630756 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.630733 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-model-cache" pod "f80d1e19-3ce6-4060-8305-7cfec3f4c296" (UID: "f80d1e19-3ce6-4060-8305-7cfec3f4c296"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:57.631323 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.631298 4855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-home\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.631323 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.631327 4855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-model-cache\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.631545 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.631344 4855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tmp-dir\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.631545 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.631358 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82rtl\" (UniqueName: \"kubernetes.io/projected/97ba4903-d54a-4291-bd8b-811bc6b3302a-kube-api-access-82rtl\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.632756 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.632730 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80d1e19-3ce6-4060-8305-7cfec3f4c296-kube-api-access-2c5f4" pod "f80d1e19-3ce6-4060-8305-7cfec3f4c296" (UID: "f80d1e19-3ce6-4060-8305-7cfec3f4c296"). InnerVolumeSpecName "kube-api-access-2c5f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:10:57.632845 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.632767 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tls-certs" pod "f80d1e19-3ce6-4060-8305-7cfec3f4c296" (UID: "f80d1e19-3ce6-4060-8305-7cfec3f4c296"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:10:57.632845 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.632815 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-dshm" pod "f80d1e19-3ce6-4060-8305-7cfec3f4c296" (UID: "f80d1e19-3ce6-4060-8305-7cfec3f4c296"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:10:57.732793 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.732763 4855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80d1e19-3ce6-4060-8305-7cfec3f4c296-tls-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.732793 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.732793 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c5f4\" (UniqueName: \"kubernetes.io/projected/f80d1e19-3ce6-4060-8305-7cfec3f4c296-kube-api-access-2c5f4\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:57.732987 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:57.732802 4855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80d1e19-3ce6-4060-8305-7cfec3f4c296-dshm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:10:58.387247 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.387210 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" path="/var/lib/kubelet/pods/0e2e8f6c-d325-4e8c-9f49-93477681daab/volumes" Nov 03 06:10:58.391495 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.390161 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299590cc-de3d-4408-8653-320887002525" path="/var/lib/kubelet/pods/299590cc-de3d-4408-8653-320887002525/volumes" Nov 03 06:10:58.395364 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.395334 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" path="/var/lib/kubelet/pods/c4fbec74-f9e1-4d46-abcc-cedad8e517fd/volumes" Nov 03 06:10:58.484363 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.484339 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" Nov 03 06:10:58.484518 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.484373 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6" event={"ID":"97ba4903-d54a-4291-bd8b-811bc6b3302a","Type":"ContainerDied","Data":"52a1543c17e2364901511c32ddd952af1ebc1cdf92a17aebc585ca329b2f16d0"} Nov 03 06:10:58.484518 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.484452 4855 scope.go:122] "RemoveContainer" containerID="e4f569de32d758da29731687c86c635b3a3f96c201349090cf278d77c08deaea" Nov 03 06:10:58.486023 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.485951 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" Nov 03 06:10:58.486023 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.485980 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn" event={"ID":"f80d1e19-3ce6-4060-8305-7cfec3f4c296","Type":"ContainerDied","Data":"400d3d57bed13e400312975f2bb918e28fcbac500ebf1ef6b36fc11f60744e19"} Nov 03 06:10:58.494381 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.494362 4855 scope.go:122] "RemoveContainer" containerID="98020769a3dbbce0f9c7eac3f06d68cc0aaffac8400e053c094707f2ff860dab" Nov 03 06:10:58.544542 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.544505 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6"] Nov 03 06:10:58.557013 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.556989 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be1a66-6d86d66b7f-2bhg6"] Nov 03 06:10:58.584578 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.584552 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn"] Nov 03 06:10:58.590425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:10:58.590398 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/distinct-w0-df7236-kserve-786cb9dbc-98rdn"] Nov 03 06:10:58.885961 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:10:58.885926 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:11:00.333582 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:00.333552 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ba4903-d54a-4291-bd8b-811bc6b3302a" path="/var/lib/kubelet/pods/97ba4903-d54a-4291-bd8b-811bc6b3302a/volumes" Nov 03 06:11:00.333997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:00.333982 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" path="/var/lib/kubelet/pods/f80d1e19-3ce6-4060-8305-7cfec3f4c296/volumes" Nov 03 06:11:02.191171 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.191093 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5"] Nov 03 06:11:02.192416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.192318 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" podUID="5809e76e-9af0-4889-bf3c-5b9b7102eb1d" containerName="maas-api" containerID="cri-o://7be8ecbbd209907c93a0ef4e24893a7ece5d1b946bc33c68d2c9a0b314e9031d" gracePeriod=30 Nov 03 06:11:02.517507 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.516407 4855 generic.go:356] "Generic (PLEG): container finished" podID="5809e76e-9af0-4889-bf3c-5b9b7102eb1d" containerID="7be8ecbbd209907c93a0ef4e24893a7ece5d1b946bc33c68d2c9a0b314e9031d" exitCode=0 Nov 03 06:11:02.517507 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.516425 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" event={"ID":"5809e76e-9af0-4889-bf3c-5b9b7102eb1d","Type":"ContainerDied","Data":"7be8ecbbd209907c93a0ef4e24893a7ece5d1b946bc33c68d2c9a0b314e9031d"} Nov 03 06:11:02.653889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.653858 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:11:02.781486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.778337 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b"] Nov 03 06:11:02.781486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.778607 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" podUID="ea38a440-6197-4e41-b316-955e76211942" containerName="maas-api" containerID="cri-o://5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c" gracePeriod=30 Nov 03 06:11:02.793124 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.793097 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-kube-api-access-8649w\" (UniqueName: \"kubernetes.io/projected/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-kube-api-access-8649w\") pod \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " Nov 03 06:11:02.793296 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.793277 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-maas-api-tls\") pod \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " Nov 03 06:11:02.793516 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.793332 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-openshift-service-ca\") pod \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " Nov 03 06:11:02.793516 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.793370 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-metrics-certs\") pod \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\" (UID: \"5809e76e-9af0-4889-bf3c-5b9b7102eb1d\") " Nov 03 06:11:02.795030 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.795003 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-openshift-service-ca" pod "5809e76e-9af0-4889-bf3c-5b9b7102eb1d" (UID: "5809e76e-9af0-4889-bf3c-5b9b7102eb1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:02.798672 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.798644 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-kube-api-access-8649w" pod "5809e76e-9af0-4889-bf3c-5b9b7102eb1d" (UID: "5809e76e-9af0-4889-bf3c-5b9b7102eb1d"). InnerVolumeSpecName "kube-api-access-8649w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:02.798957 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.798932 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-metrics-certs" pod "5809e76e-9af0-4889-bf3c-5b9b7102eb1d" (UID: "5809e76e-9af0-4889-bf3c-5b9b7102eb1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:11:02.800545 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.800509 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-maas-api-tls" pod "5809e76e-9af0-4889-bf3c-5b9b7102eb1d" (UID: "5809e76e-9af0-4889-bf3c-5b9b7102eb1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:11:02.895013 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.894983 4855 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-maas-api-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:02.895013 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.895014 4855 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-openshift-service-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:02.895206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.895028 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:02.895206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:02.895042 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8649w\" (UniqueName: \"kubernetes.io/projected/5809e76e-9af0-4889-bf3c-5b9b7102eb1d-kube-api-access-8649w\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:03.029383 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.029354 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:11:03.199803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.199771 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-maas-api-tls\") pod \"ea38a440-6197-4e41-b316-955e76211942\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " Nov 03 06:11:03.200224 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.199949 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea38a440-6197-4e41-b316-955e76211942-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea38a440-6197-4e41-b316-955e76211942-openshift-service-ca\") pod \"ea38a440-6197-4e41-b316-955e76211942\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " Nov 03 06:11:03.200224 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.199996 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-metrics-certs\") pod \"ea38a440-6197-4e41-b316-955e76211942\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " Nov 03 06:11:03.200224 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.200028 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea38a440-6197-4e41-b316-955e76211942-kube-api-access-h45qz\" (UniqueName: \"kubernetes.io/projected/ea38a440-6197-4e41-b316-955e76211942-kube-api-access-h45qz\") pod \"ea38a440-6197-4e41-b316-955e76211942\" (UID: \"ea38a440-6197-4e41-b316-955e76211942\") " Nov 03 06:11:03.200889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.200454 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea38a440-6197-4e41-b316-955e76211942-openshift-service-ca" pod "ea38a440-6197-4e41-b316-955e76211942" (UID: "ea38a440-6197-4e41-b316-955e76211942"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:03.200889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.200686 4855 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea38a440-6197-4e41-b316-955e76211942-openshift-service-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:03.202161 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.202138 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-metrics-certs" pod "ea38a440-6197-4e41-b316-955e76211942" (UID: "ea38a440-6197-4e41-b316-955e76211942"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:11:03.202230 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.202178 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-maas-api-tls" pod "ea38a440-6197-4e41-b316-955e76211942" (UID: "ea38a440-6197-4e41-b316-955e76211942"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:11:03.202265 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.202231 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea38a440-6197-4e41-b316-955e76211942-kube-api-access-h45qz" pod "ea38a440-6197-4e41-b316-955e76211942" (UID: "ea38a440-6197-4e41-b316-955e76211942"). InnerVolumeSpecName "kube-api-access-h45qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:03.320416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.320321 4855 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-maas-api-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:03.320416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.320359 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea38a440-6197-4e41-b316-955e76211942-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:03.320416 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.320373 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h45qz\" (UniqueName: \"kubernetes.io/projected/ea38a440-6197-4e41-b316-955e76211942-kube-api-access-h45qz\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:03.565294 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.565253 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" event={"ID":"5809e76e-9af0-4889-bf3c-5b9b7102eb1d","Type":"ContainerDied","Data":"b95a71e2d1aa5589c3f2705733e8c9ae642c42e96eab91e6f316357a5c6a467f"} Nov 03 06:11:03.565590 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.565569 4855 scope.go:122] "RemoveContainer" containerID="7be8ecbbd209907c93a0ef4e24893a7ece5d1b946bc33c68d2c9a0b314e9031d" Nov 03 06:11:03.572412 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.565921 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5" Nov 03 06:11:03.594019 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.585535 4855 generic.go:356] "Generic (PLEG): container finished" podID="ea38a440-6197-4e41-b316-955e76211942" containerID="5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c" exitCode=0 Nov 03 06:11:03.594019 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.585641 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" event={"ID":"ea38a440-6197-4e41-b316-955e76211942","Type":"ContainerDied","Data":"5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c"} Nov 03 06:11:03.594019 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.585691 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" event={"ID":"ea38a440-6197-4e41-b316-955e76211942","Type":"ContainerDied","Data":"28a967dd6345ee7c88bdf5f7c22b9d5f56e9bc2d3ff2ffbb89f7292b75947ffa"} Nov 03 06:11:03.594019 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.585773 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b" Nov 03 06:11:03.607082 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.605885 4855 scope.go:122] "RemoveContainer" containerID="5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c" Nov 03 06:11:03.621951 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.621928 4855 scope.go:122] "RemoveContainer" containerID="5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c" Nov 03 06:11:03.622247 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:03.622229 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c\": container with ID starting with 5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c not found: ID does not exist" containerID="5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c" Nov 03 06:11:03.622312 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.622259 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c"} err="failed to get container status \"5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c\": rpc error: code = NotFound desc = could not find container \"5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c\": container with ID starting with 5785f67dcb680e9d0755c7f20db7eb4bc0c939a8af15aeb3c557071437f7112c not found: ID does not exist" Nov 03 06:11:03.666841 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.666811 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b"] Nov 03 06:11:03.680346 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.680313 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-86728b-5854654b46-t9q8b"] Nov 03 06:11:03.721794 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.721769 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5"] Nov 03 06:11:03.728062 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:03.728034 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df7236-68fd55bc95-7t8m5"] Nov 03 06:11:04.333797 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:04.333763 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5809e76e-9af0-4889-bf3c-5b9b7102eb1d" path="/var/lib/kubelet/pods/5809e76e-9af0-4889-bf3c-5b9b7102eb1d/volumes" Nov 03 06:11:04.334268 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:04.334253 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea38a440-6197-4e41-b316-955e76211942" path="/var/lib/kubelet/pods/ea38a440-6197-4e41-b316-955e76211942/volumes" Nov 03 06:11:05.811267 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:05.811231 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf"] Nov 03 06:11:09.780148 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:09.780108 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj"] Nov 03 06:11:12.512419 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:12.512088 4855 scope.go:122] "RemoveContainer" containerID="acda9f71ce6486797dcdc558701e986441597807a6a587696c7c3ce458cfb428" Nov 03 06:11:15.200553 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:15.200518 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp"] Nov 03 06:11:15.201074 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:15.200716 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" containerID="cri-o://244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5" gracePeriod=30 Nov 03 06:11:15.215169 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:15.214711 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6"] Nov 03 06:11:15.215169 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:15.214968 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" containerID="cri-o://9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798" gracePeriod=30 Nov 03 06:11:15.283413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:15.278594 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf"] Nov 03 06:11:15.283413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:15.278850 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" containerID="cri-o://b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd" gracePeriod=30 Nov 03 06:11:16.149133 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.149097 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs"] Nov 03 06:11:16.149352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.149324 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" containerID="cri-o://f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af" gracePeriod=30 Nov 03 06:11:16.162062 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.162030 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd"] Nov 03 06:11:16.162332 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.162282 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" containerID="cri-o://e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8" gracePeriod=30 Nov 03 06:11:16.164892 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.164865 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s"] Nov 03 06:11:16.165090 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.165052 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" containerID="cri-o://fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8" gracePeriod=30 Nov 03 06:11:16.190461 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.190431 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t"] Nov 03 06:11:16.190623 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.190604 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" containerID="cri-o://917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b" gracePeriod=30 Nov 03 06:11:16.749758 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.749726 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb"] Nov 03 06:11:16.750208 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.749929 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" containerID="cri-o://8c9ac4e3dd1e867cc099dbdd318db6729f0fd37ee3058499c513675eeeb5e878" gracePeriod=30 Nov 03 06:11:16.761952 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.761922 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2"] Nov 03 06:11:16.762135 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.762116 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" containerID="cri-o://eb13878edd20bc24955729046363a55f7c55d6fd565180eba2f3df9678a623ab" gracePeriod=30 Nov 03 06:11:16.769189 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.769162 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn"] Nov 03 06:11:16.769427 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.769377 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" containerID="cri-o://c099a8b6c465176dc88af56014616d5c8cd601520b6d2832db31e643e19861ad" gracePeriod=30 Nov 03 06:11:16.780980 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.780955 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw"] Nov 03 06:11:16.781168 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.781149 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" containerID="cri-o://55491e7918d0c8346bfafcff110bb85b41026bbe975dcbc5bd01054ea3a5939f" gracePeriod=30 Nov 03 06:11:16.850733 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.850699 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:11:16.944424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.938305 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:16.944424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:16.938383 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:17.397647 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:17.397611 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:17.397845 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:17.397688 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:18.598415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:18.595612 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:18.598415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:18.595711 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:18.755572 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:18.755536 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:18.755858 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:18.755829 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:18.830414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:18.829468 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:18.830414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:18.829544 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:19.006475 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:19.005496 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5" podUID="ec588a2d-3f62-4256-afed-c86041c5a59b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:11:19.095414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:19.088491 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 06:11:19.306413 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:19.304499 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:11:20.087274 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.087238 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:20.087741 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.087298 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:20.139497 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.139463 4855 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:20.139660 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.139533 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:20.616576 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.616525 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:11:20.646561 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.646538 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:11:20.687363 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.687334 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:11:20.694288 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.694258 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-token\" (UniqueName: \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-token\") pod \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " Nov 03 06:11:20.694440 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.694313 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istiod-ca-cert\") pod \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " Nov 03 06:11:20.694440 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.694417 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-podinfo\") pod \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " Nov 03 06:11:20.694574 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.694456 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-data\") pod \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " Nov 03 06:11:20.694632 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.694572 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-certs\") pod \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " Nov 03 06:11:20.694691 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.694680 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-envoy\") pod \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " Nov 03 06:11:20.694748 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.694719 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-kube-api-access-gq9pz\" (UniqueName: \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-kube-api-access-gq9pz\") pod \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " Nov 03 06:11:20.694811 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.694792 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-credential-socket\") pod \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " Nov 03 06:11:20.694877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.694838 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-socket\") pod \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\" (UID: \"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62\") " Nov 03 06:11:20.695682 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.694786 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istiod-ca-cert" pod "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" (UID: "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:20.695682 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.695319 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-certs" pod "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" (UID: "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.695682 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.695479 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-credential-socket" pod "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" (UID: "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.695682 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.695572 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-socket" pod "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" (UID: "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.695682 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.695637 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-data" pod "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" (UID: "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.696008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.695748 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.696008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.695765 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.696008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.695779 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.696008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.695794 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.696008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.695808 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.696966 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.696942 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-token" pod "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" (UID: "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:20.697603 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.697582 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-podinfo" pod "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" (UID: "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:20.699158 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.699133 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-envoy" pod "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" (UID: "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.699271 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.699249 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-kube-api-access-gq9pz" pod "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" (UID: "ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62"). InnerVolumeSpecName "kube-api-access-gq9pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:20.803129 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803093 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-token\" (UniqueName: \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-token\") pod \"d48278d0-aa0f-4f79-af81-39f95b6c5495\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " Nov 03 06:11:20.803129 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803143 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48278d0-aa0f-4f79-af81-39f95b6c5495-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d48278d0-aa0f-4f79-af81-39f95b6c5495-istiod-ca-cert\") pod \"d48278d0-aa0f-4f79-af81-39f95b6c5495\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " Nov 03 06:11:20.803362 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803234 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-data\") pod \"ee92591f-8473-4981-8af8-bb60f096c653\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " Nov 03 06:11:20.803362 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803314 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-kube-api-access-dwhvw\" (UniqueName: \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-kube-api-access-dwhvw\") pod \"d48278d0-aa0f-4f79-af81-39f95b6c5495\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " Nov 03 06:11:20.803362 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803356 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-envoy\") pod \"d48278d0-aa0f-4f79-af81-39f95b6c5495\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " Nov 03 06:11:20.803545 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803401 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-socket\") pod \"d48278d0-aa0f-4f79-af81-39f95b6c5495\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " Nov 03 06:11:20.803545 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803434 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-istio-token\" (UniqueName: \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-istio-token\") pod \"ee92591f-8473-4981-8af8-bb60f096c653\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " Nov 03 06:11:20.803545 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803473 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-podinfo\") pod \"d48278d0-aa0f-4f79-af81-39f95b6c5495\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " Nov 03 06:11:20.803545 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803526 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-socket\") pod \"ee92591f-8473-4981-8af8-bb60f096c653\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " Nov 03 06:11:20.803990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803556 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-certs\") pod \"ee92591f-8473-4981-8af8-bb60f096c653\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " Nov 03 06:11:20.803990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803593 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee92591f-8473-4981-8af8-bb60f096c653-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee92591f-8473-4981-8af8-bb60f096c653-istiod-ca-cert\") pod \"ee92591f-8473-4981-8af8-bb60f096c653\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " Nov 03 06:11:20.803990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803631 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-credential-socket\") pod \"d48278d0-aa0f-4f79-af81-39f95b6c5495\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " Nov 03 06:11:20.803990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803661 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee92591f-8473-4981-8af8-bb60f096c653-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee92591f-8473-4981-8af8-bb60f096c653-istio-podinfo\") pod \"ee92591f-8473-4981-8af8-bb60f096c653\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " Nov 03 06:11:20.803990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803721 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-kube-api-access-5z9dt\" (UniqueName: \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-kube-api-access-5z9dt\") pod \"ee92591f-8473-4981-8af8-bb60f096c653\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " Nov 03 06:11:20.803990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803782 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-certs\") pod \"d48278d0-aa0f-4f79-af81-39f95b6c5495\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " Nov 03 06:11:20.803990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803838 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-credential-socket\") pod \"ee92591f-8473-4981-8af8-bb60f096c653\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " Nov 03 06:11:20.803990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803888 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-envoy\") pod \"ee92591f-8473-4981-8af8-bb60f096c653\" (UID: \"ee92591f-8473-4981-8af8-bb60f096c653\") " Nov 03 06:11:20.803990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803982 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-data\") pod \"d48278d0-aa0f-4f79-af81-39f95b6c5495\" (UID: \"d48278d0-aa0f-4f79-af81-39f95b6c5495\") " Nov 03 06:11:20.804501 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.803983 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-socket" pod "d48278d0-aa0f-4f79-af81-39f95b6c5495" (UID: "d48278d0-aa0f-4f79-af81-39f95b6c5495"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.804501 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.804318 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-data" pod "ee92591f-8473-4981-8af8-bb60f096c653" (UID: "ee92591f-8473-4981-8af8-bb60f096c653"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.804932 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-data" pod "d48278d0-aa0f-4f79-af81-39f95b6c5495" (UID: "d48278d0-aa0f-4f79-af81-39f95b6c5495"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.804956 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-credential-socket" pod "d48278d0-aa0f-4f79-af81-39f95b6c5495" (UID: "d48278d0-aa0f-4f79-af81-39f95b6c5495"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.805358 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48278d0-aa0f-4f79-af81-39f95b6c5495-istiod-ca-cert" pod "d48278d0-aa0f-4f79-af81-39f95b6c5495" (UID: "d48278d0-aa0f-4f79-af81-39f95b6c5495"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.805818 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.805841 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.805857 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.805873 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.805886 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.805901 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d48278d0-aa0f-4f79-af81-39f95b6c5495-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.805914 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.805926 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.805941 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9pz\" (UniqueName: \"kubernetes.io/projected/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62-kube-api-access-gq9pz\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.806131 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-certs" pod "d48278d0-aa0f-4f79-af81-39f95b6c5495" (UID: "d48278d0-aa0f-4f79-af81-39f95b6c5495"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.806421 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-credential-socket" pod "ee92591f-8473-4981-8af8-bb60f096c653" (UID: "ee92591f-8473-4981-8af8-bb60f096c653"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.806631 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-socket" pod "ee92591f-8473-4981-8af8-bb60f096c653" (UID: "ee92591f-8473-4981-8af8-bb60f096c653"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.817656 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee92591f-8473-4981-8af8-bb60f096c653-istio-podinfo" pod "ee92591f-8473-4981-8af8-bb60f096c653" (UID: "ee92591f-8473-4981-8af8-bb60f096c653"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.817927 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-certs" pod "ee92591f-8473-4981-8af8-bb60f096c653" (UID: "ee92591f-8473-4981-8af8-bb60f096c653"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.818209 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee92591f-8473-4981-8af8-bb60f096c653-istiod-ca-cert" pod "ee92591f-8473-4981-8af8-bb60f096c653" (UID: "ee92591f-8473-4981-8af8-bb60f096c653"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.827050 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-podinfo" pod "d48278d0-aa0f-4f79-af81-39f95b6c5495" (UID: "d48278d0-aa0f-4f79-af81-39f95b6c5495"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.827289 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-envoy" pod "ee92591f-8473-4981-8af8-bb60f096c653" (UID: "ee92591f-8473-4981-8af8-bb60f096c653"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.827434 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-kube-api-access-5z9dt" pod "ee92591f-8473-4981-8af8-bb60f096c653" (UID: "ee92591f-8473-4981-8af8-bb60f096c653"). InnerVolumeSpecName "kube-api-access-5z9dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.827534 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-envoy" pod "d48278d0-aa0f-4f79-af81-39f95b6c5495" (UID: "d48278d0-aa0f-4f79-af81-39f95b6c5495"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:20.832486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.827627 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-istio-token" pod "ee92591f-8473-4981-8af8-bb60f096c653" (UID: "ee92591f-8473-4981-8af8-bb60f096c653"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:20.866696 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.841569 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-kube-api-access-dwhvw" pod "d48278d0-aa0f-4f79-af81-39f95b6c5495" (UID: "d48278d0-aa0f-4f79-af81-39f95b6c5495"). InnerVolumeSpecName "kube-api-access-dwhvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:20.866696 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.850981 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-token" pod "d48278d0-aa0f-4f79-af81-39f95b6c5495" (UID: "d48278d0-aa0f-4f79-af81-39f95b6c5495"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:20.866696 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.854980 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" containerID="cri-o://afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc" gracePeriod=45 Nov 03 06:11:20.907103 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907073 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhvw\" (UniqueName: \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-kube-api-access-dwhvw\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907103 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907102 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907114 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907122 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907130 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907139 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907149 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee92591f-8473-4981-8af8-bb60f096c653-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907157 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee92591f-8473-4981-8af8-bb60f096c653-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907165 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z9dt\" (UniqueName: \"kubernetes.io/projected/ee92591f-8473-4981-8af8-bb60f096c653-kube-api-access-5z9dt\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907174 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d48278d0-aa0f-4f79-af81-39f95b6c5495-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907183 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907193 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee92591f-8473-4981-8af8-bb60f096c653-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.907295 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.907202 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d48278d0-aa0f-4f79-af81-39f95b6c5495-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:20.916756 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.916727 4855 generic.go:356] "Generic (PLEG): container finished" podID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerID="9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798" exitCode=0 Nov 03 06:11:20.916963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.916796 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" Nov 03 06:11:20.916963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.916835 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" event={"ID":"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62","Type":"ContainerDied","Data":"9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798"} Nov 03 06:11:20.916963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.916873 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" event={"ID":"ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62","Type":"ContainerDied","Data":"9ed8fb493e9ed3f246fc92483dcc32c2cf909d259a3eb5a8037309af3eeaaa04"} Nov 03 06:11:20.916963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.916895 4855 scope.go:122] "RemoveContainer" containerID="9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798" Nov 03 06:11:20.918776 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.918753 4855 generic.go:356] "Generic (PLEG): container finished" podID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerID="b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd" exitCode=0 Nov 03 06:11:20.918892 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.918801 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" Nov 03 06:11:20.918892 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.918832 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" event={"ID":"d48278d0-aa0f-4f79-af81-39f95b6c5495","Type":"ContainerDied","Data":"b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd"} Nov 03 06:11:20.918892 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.918874 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf" event={"ID":"d48278d0-aa0f-4f79-af81-39f95b6c5495","Type":"ContainerDied","Data":"93ffe2fa749ff4a4752abd6549132e2d7b4b1c8787eca1872e4016c4da078eb7"} Nov 03 06:11:20.929106 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.920265 4855 generic.go:356] "Generic (PLEG): container finished" podID="ee92591f-8473-4981-8af8-bb60f096c653" containerID="244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5" exitCode=0 Nov 03 06:11:20.929106 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.920307 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" event={"ID":"ee92591f-8473-4981-8af8-bb60f096c653","Type":"ContainerDied","Data":"244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5"} Nov 03 06:11:20.929106 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.920317 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" Nov 03 06:11:20.929106 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.920337 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp" event={"ID":"ee92591f-8473-4981-8af8-bb60f096c653","Type":"ContainerDied","Data":"7c9527be9866fdfc4c423647ff623f38874eee25a662f2548d425518bd1c78d6"} Nov 03 06:11:20.975322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.975302 4855 scope.go:122] "RemoveContainer" containerID="9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798" Nov 03 06:11:20.975711 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:20.975682 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798\": container with ID starting with 9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798 not found: ID does not exist" containerID="9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798" Nov 03 06:11:20.975831 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.975719 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798"} err="failed to get container status \"9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798\": rpc error: code = NotFound desc = could not find container \"9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798\": container with ID starting with 9773ec246a48227754557cb828910431f8e9fc3e15842b7c5660bccbdfa35798 not found: ID does not exist" Nov 03 06:11:20.975831 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.975737 4855 scope.go:122] "RemoveContainer" containerID="b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd" Nov 03 06:11:20.984503 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.984481 4855 scope.go:122] "RemoveContainer" containerID="b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd" Nov 03 06:11:20.984771 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:20.984749 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd\": container with ID starting with b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd not found: ID does not exist" containerID="b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd" Nov 03 06:11:20.984862 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.984778 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd"} err="failed to get container status \"b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd\": rpc error: code = NotFound desc = could not find container \"b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd\": container with ID starting with b951ee87268d298b55c763191f721d908c5e57a79bed69d724b23b5eab5508dd not found: ID does not exist" Nov 03 06:11:20.984862 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.984800 4855 scope.go:122] "RemoveContainer" containerID="244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5" Nov 03 06:11:20.995918 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.995896 4855 scope.go:122] "RemoveContainer" containerID="244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5" Nov 03 06:11:20.996332 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:20.996299 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5\": container with ID starting with 244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5 not found: ID does not exist" containerID="244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5" Nov 03 06:11:20.996332 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:20.996327 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5"} err="failed to get container status \"244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5\": rpc error: code = NotFound desc = could not find container \"244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5\": container with ID starting with 244259db9648de48dbcc1188750f4c7446f5b244795e9ff9f727557b3dd3a8a5 not found: ID does not exist" Nov 03 06:11:21.015122 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.015072 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6"] Nov 03 06:11:21.027538 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.027503 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6"] Nov 03 06:11:21.142469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.137594 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp"] Nov 03 06:11:21.142469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.141239 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-8t5dp"] Nov 03 06:11:21.181238 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.181174 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf"] Nov 03 06:11:21.185776 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.185733 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-x55jf"] Nov 03 06:11:21.423988 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.423908 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:11:21.499740 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.499703 4855 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:11:21.499882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.499767 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-z9mk6" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:11:21.561053 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.561010 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-model-cache\") pod \"8e65c31d-5628-4811-a84e-405aa9eb9188\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " Nov 03 06:11:21.561233 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.561185 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-dshm\") pod \"8e65c31d-5628-4811-a84e-405aa9eb9188\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " Nov 03 06:11:21.561309 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.561248 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-tmp-dir\") pod \"8e65c31d-5628-4811-a84e-405aa9eb9188\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " Nov 03 06:11:21.561309 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.561299 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e65c31d-5628-4811-a84e-405aa9eb9188-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e65c31d-5628-4811-a84e-405aa9eb9188-tls-certs\") pod \"8e65c31d-5628-4811-a84e-405aa9eb9188\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " Nov 03 06:11:21.561577 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.561310 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-model-cache" pod "8e65c31d-5628-4811-a84e-405aa9eb9188" (UID: "8e65c31d-5628-4811-a84e-405aa9eb9188"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.561577 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.561325 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-home\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-home\") pod \"8e65c31d-5628-4811-a84e-405aa9eb9188\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " Nov 03 06:11:21.561878 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.561852 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e65c31d-5628-4811-a84e-405aa9eb9188-kube-api-access-2nz7h\" (UniqueName: \"kubernetes.io/projected/8e65c31d-5628-4811-a84e-405aa9eb9188-kube-api-access-2nz7h\") pod \"8e65c31d-5628-4811-a84e-405aa9eb9188\" (UID: \"8e65c31d-5628-4811-a84e-405aa9eb9188\") " Nov 03 06:11:21.562315 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.562296 4855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-model-cache\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.567905 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.567870 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-dshm" pod "8e65c31d-5628-4811-a84e-405aa9eb9188" (UID: "8e65c31d-5628-4811-a84e-405aa9eb9188"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.570636 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.570612 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-home" pod "8e65c31d-5628-4811-a84e-405aa9eb9188" (UID: "8e65c31d-5628-4811-a84e-405aa9eb9188"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.570971 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.570944 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-tmp-dir" pod "8e65c31d-5628-4811-a84e-405aa9eb9188" (UID: "8e65c31d-5628-4811-a84e-405aa9eb9188"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.572963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.572929 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e65c31d-5628-4811-a84e-405aa9eb9188-tls-certs" pod "8e65c31d-5628-4811-a84e-405aa9eb9188" (UID: "8e65c31d-5628-4811-a84e-405aa9eb9188"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:11:21.575353 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.575329 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e65c31d-5628-4811-a84e-405aa9eb9188-kube-api-access-2nz7h" pod "8e65c31d-5628-4811-a84e-405aa9eb9188" (UID: "8e65c31d-5628-4811-a84e-405aa9eb9188"). InnerVolumeSpecName "kube-api-access-2nz7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:21.616326 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.616299 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:11:21.618820 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.618799 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:11:21.663281 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663251 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-data\") pod \"cafef133-515b-4fdf-a799-cf10a42224f8\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " Nov 03 06:11:21.663478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663291 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-credential-socket\") pod \"cafef133-515b-4fdf-a799-cf10a42224f8\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " Nov 03 06:11:21.663478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663312 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-socket\") pod \"d926084c-dafb-4fd7-adc3-928dc9036a92\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " Nov 03 06:11:21.663478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663334 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-data\") pod \"d926084c-dafb-4fd7-adc3-928dc9036a92\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " Nov 03 06:11:21.663478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663361 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-certs\") pod \"cafef133-515b-4fdf-a799-cf10a42224f8\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " Nov 03 06:11:21.663478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663447 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-kube-api-access-lx78j\" (UniqueName: \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-kube-api-access-lx78j\") pod \"d926084c-dafb-4fd7-adc3-928dc9036a92\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663506 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-envoy\") pod \"cafef133-515b-4fdf-a799-cf10a42224f8\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663545 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-certs\") pod \"d926084c-dafb-4fd7-adc3-928dc9036a92\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663581 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-credential-socket\") pod \"d926084c-dafb-4fd7-adc3-928dc9036a92\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663592 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-data" pod "cafef133-515b-4fdf-a799-cf10a42224f8" (UID: "cafef133-515b-4fdf-a799-cf10a42224f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663611 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-credential-socket" pod "cafef133-515b-4fdf-a799-cf10a42224f8" (UID: "cafef133-515b-4fdf-a799-cf10a42224f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663675 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-envoy\") pod \"d926084c-dafb-4fd7-adc3-928dc9036a92\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663690 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-socket" pod "d926084c-dafb-4fd7-adc3-928dc9036a92" (UID: "d926084c-dafb-4fd7-adc3-928dc9036a92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663709 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-data" pod "d926084c-dafb-4fd7-adc3-928dc9036a92" (UID: "d926084c-dafb-4fd7-adc3-928dc9036a92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663712 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cafef133-515b-4fdf-a799-cf10a42224f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafef133-515b-4fdf-a799-cf10a42224f8-istio-podinfo\") pod \"cafef133-515b-4fdf-a799-cf10a42224f8\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663694 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-certs" pod "cafef133-515b-4fdf-a799-cf10a42224f8" (UID: "cafef133-515b-4fdf-a799-cf10a42224f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.663837 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663833 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafef133-515b-4fdf-a799-cf10a42224f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafef133-515b-4fdf-a799-cf10a42224f8-istiod-ca-cert\") pod \"cafef133-515b-4fdf-a799-cf10a42224f8\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " Nov 03 06:11:21.664358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663847 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-certs" pod "d926084c-dafb-4fd7-adc3-928dc9036a92" (UID: "d926084c-dafb-4fd7-adc3-928dc9036a92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.664358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663880 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-istio-token\" (UniqueName: \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-istio-token\") pod \"cafef133-515b-4fdf-a799-cf10a42224f8\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " Nov 03 06:11:21.664358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663889 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-credential-socket" pod "d926084c-dafb-4fd7-adc3-928dc9036a92" (UID: "d926084c-dafb-4fd7-adc3-928dc9036a92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.664358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663913 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-podinfo\") pod \"d926084c-dafb-4fd7-adc3-928dc9036a92\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " Nov 03 06:11:21.664358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.663953 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d926084c-dafb-4fd7-adc3-928dc9036a92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d926084c-dafb-4fd7-adc3-928dc9036a92-istiod-ca-cert\") pod \"d926084c-dafb-4fd7-adc3-928dc9036a92\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " Nov 03 06:11:21.664358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664006 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-token\" (UniqueName: \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-token\") pod \"d926084c-dafb-4fd7-adc3-928dc9036a92\" (UID: \"d926084c-dafb-4fd7-adc3-928dc9036a92\") " Nov 03 06:11:21.664358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664034 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-socket\") pod \"cafef133-515b-4fdf-a799-cf10a42224f8\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " Nov 03 06:11:21.664358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664069 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-kube-api-access-btsk7\" (UniqueName: \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-kube-api-access-btsk7\") pod \"cafef133-515b-4fdf-a799-cf10a42224f8\" (UID: \"cafef133-515b-4fdf-a799-cf10a42224f8\") " Nov 03 06:11:21.664358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664178 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafef133-515b-4fdf-a799-cf10a42224f8-istiod-ca-cert" pod "cafef133-515b-4fdf-a799-cf10a42224f8" (UID: "cafef133-515b-4fdf-a799-cf10a42224f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:21.664791 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664664 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.664791 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664688 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.664791 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664704 4855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-dshm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.664791 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664741 4855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-tmp-dir\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.664791 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664755 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafef133-515b-4fdf-a799-cf10a42224f8-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.664791 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664769 4855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e65c31d-5628-4811-a84e-405aa9eb9188-tls-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.664791 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664782 4855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e65c31d-5628-4811-a84e-405aa9eb9188-home\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.665132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664794 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nz7h\" (UniqueName: \"kubernetes.io/projected/8e65c31d-5628-4811-a84e-405aa9eb9188-kube-api-access-2nz7h\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.665132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664806 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.665132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664821 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.665132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664834 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.665132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664847 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.665132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664860 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.665132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.664664 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d926084c-dafb-4fd7-adc3-928dc9036a92-istiod-ca-cert" pod "d926084c-dafb-4fd7-adc3-928dc9036a92" (UID: "d926084c-dafb-4fd7-adc3-928dc9036a92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:21.665132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.665007 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-socket" pod "cafef133-515b-4fdf-a799-cf10a42224f8" (UID: "cafef133-515b-4fdf-a799-cf10a42224f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.666353 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.666325 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cafef133-515b-4fdf-a799-cf10a42224f8-istio-podinfo" pod "cafef133-515b-4fdf-a799-cf10a42224f8" (UID: "cafef133-515b-4fdf-a799-cf10a42224f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:21.666537 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.666515 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-kube-api-access-lx78j" pod "d926084c-dafb-4fd7-adc3-928dc9036a92" (UID: "d926084c-dafb-4fd7-adc3-928dc9036a92"). InnerVolumeSpecName "kube-api-access-lx78j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:21.666642 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.666625 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-envoy" pod "d926084c-dafb-4fd7-adc3-928dc9036a92" (UID: "d926084c-dafb-4fd7-adc3-928dc9036a92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.666703 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.666687 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-istio-token" pod "cafef133-515b-4fdf-a799-cf10a42224f8" (UID: "cafef133-515b-4fdf-a799-cf10a42224f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:21.666940 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.666910 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-envoy" pod "cafef133-515b-4fdf-a799-cf10a42224f8" (UID: "cafef133-515b-4fdf-a799-cf10a42224f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.667155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.667138 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-kube-api-access-btsk7" pod "cafef133-515b-4fdf-a799-cf10a42224f8" (UID: "cafef133-515b-4fdf-a799-cf10a42224f8"). InnerVolumeSpecName "kube-api-access-btsk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:21.667230 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.667146 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-podinfo" pod "d926084c-dafb-4fd7-adc3-928dc9036a92" (UID: "d926084c-dafb-4fd7-adc3-928dc9036a92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:21.667646 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.667626 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-token" pod "d926084c-dafb-4fd7-adc3-928dc9036a92" (UID: "d926084c-dafb-4fd7-adc3-928dc9036a92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:21.767178 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.766997 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx78j\" (UniqueName: \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-kube-api-access-lx78j\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.767178 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.767032 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.767178 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.767045 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.767178 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.767056 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafef133-515b-4fdf-a799-cf10a42224f8-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.767178 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.767071 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.767178 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.767083 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.767178 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.767094 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d926084c-dafb-4fd7-adc3-928dc9036a92-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.767178 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.767108 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d926084c-dafb-4fd7-adc3-928dc9036a92-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.767178 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.767121 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafef133-515b-4fdf-a799-cf10a42224f8-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.767178 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.767134 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btsk7\" (UniqueName: \"kubernetes.io/projected/cafef133-515b-4fdf-a799-cf10a42224f8-kube-api-access-btsk7\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:21.939982 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.939485 4855 generic.go:356] "Generic (PLEG): container finished" podID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerID="eb13878edd20bc24955729046363a55f7c55d6fd565180eba2f3df9678a623ab" exitCode=0 Nov 03 06:11:21.939982 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.939587 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" event={"ID":"6b19d23e-3efa-4c72-a703-6610bd96a3c4","Type":"ContainerDied","Data":"eb13878edd20bc24955729046363a55f7c55d6fd565180eba2f3df9678a623ab"} Nov 03 06:11:21.947877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.947601 4855 generic.go:356] "Generic (PLEG): container finished" podID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerID="55491e7918d0c8346bfafcff110bb85b41026bbe975dcbc5bd01054ea3a5939f" exitCode=0 Nov 03 06:11:21.947877 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.947692 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" event={"ID":"8f7ea471-1f77-49c0-92dc-139f56db1d14","Type":"ContainerDied","Data":"55491e7918d0c8346bfafcff110bb85b41026bbe975dcbc5bd01054ea3a5939f"} Nov 03 06:11:21.953041 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.953010 4855 generic.go:356] "Generic (PLEG): container finished" podID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerID="afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc" exitCode=0 Nov 03 06:11:21.953163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.953090 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" event={"ID":"8e65c31d-5628-4811-a84e-405aa9eb9188","Type":"ContainerDied","Data":"afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc"} Nov 03 06:11:21.953163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.953120 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" event={"ID":"8e65c31d-5628-4811-a84e-405aa9eb9188","Type":"ContainerDied","Data":"f6d9b0874ef2d39f33dce7d09516fa1c9c4134d896985a67fe80d8528fe50e19"} Nov 03 06:11:21.953163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.953143 4855 scope.go:122] "RemoveContainer" containerID="afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc" Nov 03 06:11:21.953331 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.953293 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" Nov 03 06:11:21.957865 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.954823 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:11:21.965033 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.964042 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:11:21.968903 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.968842 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-socket\") pod \"21990fba-a699-4ba0-8a0d-ac9025d3c129\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " Nov 03 06:11:21.969015 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.968905 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-certs\") pod \"21990fba-a699-4ba0-8a0d-ac9025d3c129\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " Nov 03 06:11:21.969015 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.968945 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21990fba-a699-4ba0-8a0d-ac9025d3c129-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21990fba-a699-4ba0-8a0d-ac9025d3c129-istiod-ca-cert\") pod \"21990fba-a699-4ba0-8a0d-ac9025d3c129\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " Nov 03 06:11:21.969015 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.968970 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-envoy\") pod \"21990fba-a699-4ba0-8a0d-ac9025d3c129\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " Nov 03 06:11:21.969194 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969035 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-credential-socket\") pod \"21990fba-a699-4ba0-8a0d-ac9025d3c129\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " Nov 03 06:11:21.969194 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969069 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-kube-api-access-2wsgl\" (UniqueName: \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-kube-api-access-2wsgl\") pod \"21990fba-a699-4ba0-8a0d-ac9025d3c129\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " Nov 03 06:11:21.969194 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969100 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istiod-ca-cert\") pod \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " Nov 03 06:11:21.969194 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969151 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-token\" (UniqueName: \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-token\") pod \"21990fba-a699-4ba0-8a0d-ac9025d3c129\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " Nov 03 06:11:21.969411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969198 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-envoy\") pod \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " Nov 03 06:11:21.969411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969238 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-podinfo\") pod \"21990fba-a699-4ba0-8a0d-ac9025d3c129\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " Nov 03 06:11:21.969411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969272 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-data\") pod \"21990fba-a699-4ba0-8a0d-ac9025d3c129\" (UID: \"21990fba-a699-4ba0-8a0d-ac9025d3c129\") " Nov 03 06:11:21.969411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969310 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-credential-socket\") pod \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " Nov 03 06:11:21.969411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969365 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-certs\") pod \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " Nov 03 06:11:21.969666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969418 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-token\" (UniqueName: \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-token\") pod \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " Nov 03 06:11:21.969666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969456 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-podinfo\") pod \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " Nov 03 06:11:21.969666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969512 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-socket\") pod \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " Nov 03 06:11:21.969666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969561 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-data\") pod \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " Nov 03 06:11:21.969666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.969599 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-kube-api-access-qfkz8\" (UniqueName: \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-kube-api-access-qfkz8\") pod \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\" (UID: \"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c\") " Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.970949 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-data" pod "21990fba-a699-4ba0-8a0d-ac9025d3c129" (UID: "21990fba-a699-4ba0-8a0d-ac9025d3c129"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.972094 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-certs" pod "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" (UID: "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.972300 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-credential-socket" pod "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" (UID: "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.972766 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-kube-api-access-qfkz8" pod "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" (UID: "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c"). InnerVolumeSpecName "kube-api-access-qfkz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.975492 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-token" pod "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" (UID: "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.975828 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-podinfo" pod "21990fba-a699-4ba0-8a0d-ac9025d3c129" (UID: "21990fba-a699-4ba0-8a0d-ac9025d3c129"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.976025 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-data" pod "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" (UID: "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.976105 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-socket" pod "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" (UID: "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.976323 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-certs" pod "21990fba-a699-4ba0-8a0d-ac9025d3c129" (UID: "21990fba-a699-4ba0-8a0d-ac9025d3c129"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.976491 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-envoy" pod "21990fba-a699-4ba0-8a0d-ac9025d3c129" (UID: "21990fba-a699-4ba0-8a0d-ac9025d3c129"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.976664 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-socket" pod "21990fba-a699-4ba0-8a0d-ac9025d3c129" (UID: "21990fba-a699-4ba0-8a0d-ac9025d3c129"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.976761 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21990fba-a699-4ba0-8a0d-ac9025d3c129-istiod-ca-cert" pod "21990fba-a699-4ba0-8a0d-ac9025d3c129" (UID: "21990fba-a699-4ba0-8a0d-ac9025d3c129"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.977042 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-credential-socket" pod "21990fba-a699-4ba0-8a0d-ac9025d3c129" (UID: "21990fba-a699-4ba0-8a0d-ac9025d3c129"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.977095 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istiod-ca-cert" pod "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" (UID: "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.979097 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-token" pod "21990fba-a699-4ba0-8a0d-ac9025d3c129" (UID: "21990fba-a699-4ba0-8a0d-ac9025d3c129"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.983381 4855 generic.go:356] "Generic (PLEG): container finished" podID="82859956-f5dd-43be-894b-2e78907f7505" containerID="c099a8b6c465176dc88af56014616d5c8cd601520b6d2832db31e643e19861ad" exitCode=0 Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.983476 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" event={"ID":"82859956-f5dd-43be-894b-2e78907f7505","Type":"ContainerDied","Data":"c099a8b6c465176dc88af56014616d5c8cd601520b6d2832db31e643e19861ad"} Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.983594 4855 scope.go:122] "RemoveContainer" containerID="afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:21.984443 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc\": container with ID starting with afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc not found: ID does not exist" containerID="afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc" Nov 03 06:11:21.986978 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.984478 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc"} err="failed to get container status \"afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc\": rpc error: code = NotFound desc = could not find container \"afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc\": container with ID starting with afe4aa5f925dc80e15ddfaae99c0856500dd0c379a538c4b08663e911eda71dc not found: ID does not exist" Nov 03 06:11:21.988419 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.988065 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-envoy" pod "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" (UID: "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:21.988745 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.988713 4855 generic.go:356] "Generic (PLEG): container finished" podID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerID="8c9ac4e3dd1e867cc099dbdd318db6729f0fd37ee3058499c513675eeeb5e878" exitCode=0 Nov 03 06:11:21.988859 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.988816 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" event={"ID":"ec3a0b0e-10e5-4605-aa0e-67745585646d","Type":"ContainerDied","Data":"8c9ac4e3dd1e867cc099dbdd318db6729f0fd37ee3058499c513675eeeb5e878"} Nov 03 06:11:21.989568 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.989469 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-kube-api-access-2wsgl" pod "21990fba-a699-4ba0-8a0d-ac9025d3c129" (UID: "21990fba-a699-4ba0-8a0d-ac9025d3c129"). InnerVolumeSpecName "kube-api-access-2wsgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:21.990477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.990451 4855 generic.go:356] "Generic (PLEG): container finished" podID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerID="f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af" exitCode=0 Nov 03 06:11:21.990477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.990529 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" event={"ID":"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c","Type":"ContainerDied","Data":"f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af"} Nov 03 06:11:21.990477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.990571 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" event={"ID":"6c5c4df2-55ce-4bd3-b10d-10ba796dce4c","Type":"ContainerDied","Data":"e34fd5cd144718ee3cea995f9cbed5e63d6edd0c250b1916934d1f2fc840117e"} Nov 03 06:11:21.990477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.990600 4855 scope.go:122] "RemoveContainer" containerID="f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af" Nov 03 06:11:21.990477 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.990720 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs" Nov 03 06:11:21.992502 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.992362 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-podinfo" pod "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" (UID: "6c5c4df2-55ce-4bd3-b10d-10ba796dce4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:21.993541 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.993517 4855 generic.go:356] "Generic (PLEG): container finished" podID="cafef133-515b-4fdf-a799-cf10a42224f8" containerID="e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8" exitCode=0 Nov 03 06:11:21.993613 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.993591 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" event={"ID":"cafef133-515b-4fdf-a799-cf10a42224f8","Type":"ContainerDied","Data":"e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8"} Nov 03 06:11:21.993676 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.993621 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" event={"ID":"cafef133-515b-4fdf-a799-cf10a42224f8","Type":"ContainerDied","Data":"b1fd3d7aa981b4b3dd068c5a4d2f51774b9c91377a00e0e0206e6ee18f0979cd"} Nov 03 06:11:21.993736 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.993677 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd" Nov 03 06:11:22.001091 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:21.999993 4855 generic.go:356] "Generic (PLEG): container finished" podID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerID="917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b" exitCode=0 Nov 03 06:11:22.001091 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.000064 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" event={"ID":"d926084c-dafb-4fd7-adc3-928dc9036a92","Type":"ContainerDied","Data":"917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b"} Nov 03 06:11:22.001091 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.000103 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" event={"ID":"d926084c-dafb-4fd7-adc3-928dc9036a92","Type":"ContainerDied","Data":"1c7a4b345b3a71b47dfa6ac3a5b02c7178f210995fa0d279f1645aa0cb9cc7aa"} Nov 03 06:11:22.001091 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.000183 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t" Nov 03 06:11:22.006422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.003184 4855 generic.go:356] "Generic (PLEG): container finished" podID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerID="fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8" exitCode=0 Nov 03 06:11:22.006422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.003238 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" event={"ID":"21990fba-a699-4ba0-8a0d-ac9025d3c129","Type":"ContainerDied","Data":"fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8"} Nov 03 06:11:22.006422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.003295 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" event={"ID":"21990fba-a699-4ba0-8a0d-ac9025d3c129","Type":"ContainerDied","Data":"ec3762b4385877197c61e1fe4a475ac2199dcb7f33ab188674bad7cc35029c86"} Nov 03 06:11:22.006422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.003365 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s" Nov 03 06:11:22.039600 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.039573 4855 scope.go:122] "RemoveContainer" containerID="f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af" Nov 03 06:11:22.040164 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:22.040136 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af\": container with ID starting with f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af not found: ID does not exist" containerID="f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af" Nov 03 06:11:22.040256 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.040176 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af"} err="failed to get container status \"f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af\": rpc error: code = NotFound desc = could not find container \"f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af\": container with ID starting with f9010e2a3d3a946f6a072afaf4e1b9cf873a6e0a1f52452ea7252aeccb1f02af not found: ID does not exist" Nov 03 06:11:22.040256 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.040201 4855 scope.go:122] "RemoveContainer" containerID="e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8" Nov 03 06:11:22.051549 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.051472 4855 scope.go:122] "RemoveContainer" containerID="e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8" Nov 03 06:11:22.051911 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:22.051891 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8\": container with ID starting with e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8 not found: ID does not exist" containerID="e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8" Nov 03 06:11:22.052091 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.052005 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8"} err="failed to get container status \"e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8\": rpc error: code = NotFound desc = could not find container \"e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8\": container with ID starting with e404da2eb4e0b7f2522d8d97f36f04507123b79cad5ff02498d3c60ba1ac1db8 not found: ID does not exist" Nov 03 06:11:22.052091 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.052026 4855 scope.go:122] "RemoveContainer" containerID="917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b" Nov 03 06:11:22.060384 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.060357 4855 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 06:11:22.060569 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.060469 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 06:11:22.063131 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.063107 4855 scope.go:122] "RemoveContainer" containerID="4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218" Nov 03 06:11:22.068804 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.068612 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:11:22.071301 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.071027 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-kube-api-access-mmsjx\" (UniqueName: \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-kube-api-access-mmsjx\") pod \"ec3a0b0e-10e5-4605-aa0e-67745585646d\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " Nov 03 06:11:22.071301 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.071172 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-certs\") pod \"ec3a0b0e-10e5-4605-aa0e-67745585646d\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " Nov 03 06:11:22.071301 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.071211 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-data\") pod \"ec3a0b0e-10e5-4605-aa0e-67745585646d\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " Nov 03 06:11:22.071301 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.071247 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-socket\") pod \"ec3a0b0e-10e5-4605-aa0e-67745585646d\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " Nov 03 06:11:22.071842 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.071275 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-credential-socket\") pod \"ec3a0b0e-10e5-4605-aa0e-67745585646d\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " Nov 03 06:11:22.072662 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.071961 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-podinfo\") pod \"ec3a0b0e-10e5-4605-aa0e-67745585646d\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " Nov 03 06:11:22.072662 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.072009 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-envoy\") pod \"ec3a0b0e-10e5-4605-aa0e-67745585646d\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " Nov 03 06:11:22.072662 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.072073 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3a0b0e-10e5-4605-aa0e-67745585646d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3a0b0e-10e5-4605-aa0e-67745585646d-istiod-ca-cert\") pod \"ec3a0b0e-10e5-4605-aa0e-67745585646d\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " Nov 03 06:11:22.072662 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.072150 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-token\" (UniqueName: \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-token\") pod \"ec3a0b0e-10e5-4605-aa0e-67745585646d\" (UID: \"ec3a0b0e-10e5-4605-aa0e-67745585646d\") " Nov 03 06:11:22.073163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073139 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073170 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073184 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073197 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073210 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073224 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073238 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073252 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073266 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073278 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073292 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfkz8\" (UniqueName: \"kubernetes.io/projected/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-kube-api-access-qfkz8\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073305 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073318 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073331 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21990fba-a699-4ba0-8a0d-ac9025d3c129-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073343 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073356 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21990fba-a699-4ba0-8a0d-ac9025d3c129-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073369 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wsgl\" (UniqueName: \"kubernetes.io/projected/21990fba-a699-4ba0-8a0d-ac9025d3c129-kube-api-access-2wsgl\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.073371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.073382 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.077751 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.077717 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Nov 03 06:11:22.077844 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.077801 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-token" pod "ec3a0b0e-10e5-4605-aa0e-67745585646d" (UID: "ec3a0b0e-10e5-4605-aa0e-67745585646d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:22.078052 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.078024 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-credential-socket" pod "ec3a0b0e-10e5-4605-aa0e-67745585646d" (UID: "ec3a0b0e-10e5-4605-aa0e-67745585646d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.078256 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.078229 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-certs" pod "ec3a0b0e-10e5-4605-aa0e-67745585646d" (UID: "ec3a0b0e-10e5-4605-aa0e-67745585646d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.078448 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.078428 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-data" pod "ec3a0b0e-10e5-4605-aa0e-67745585646d" (UID: "ec3a0b0e-10e5-4605-aa0e-67745585646d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.078641 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.078615 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-socket" pod "ec3a0b0e-10e5-4605-aa0e-67745585646d" (UID: "ec3a0b0e-10e5-4605-aa0e-67745585646d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.078912 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.078890 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3a0b0e-10e5-4605-aa0e-67745585646d-istiod-ca-cert" pod "ec3a0b0e-10e5-4605-aa0e-67745585646d" (UID: "ec3a0b0e-10e5-4605-aa0e-67745585646d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:22.079019 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.079000 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-kube-api-access-mmsjx" pod "ec3a0b0e-10e5-4605-aa0e-67745585646d" (UID: "ec3a0b0e-10e5-4605-aa0e-67745585646d"). InnerVolumeSpecName "kube-api-access-mmsjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:22.080492 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.080470 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-podinfo" pod "ec3a0b0e-10e5-4605-aa0e-67745585646d" (UID: "ec3a0b0e-10e5-4605-aa0e-67745585646d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:22.080902 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.080878 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-envoy" pod "ec3a0b0e-10e5-4605-aa0e-67745585646d" (UID: "ec3a0b0e-10e5-4605-aa0e-67745585646d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.081230 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.081210 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf"] Nov 03 06:11:22.087498 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.087380 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/premium-w2-5c3542-kserve-6666fb747d-9vwwf"] Nov 03 06:11:22.101937 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.101905 4855 scope.go:122] "RemoveContainer" containerID="917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b" Nov 03 06:11:22.102447 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:22.102416 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b\": container with ID starting with 917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b not found: ID does not exist" containerID="917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b" Nov 03 06:11:22.102550 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.102456 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b"} err="failed to get container status \"917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b\": rpc error: code = NotFound desc = could not find container \"917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b\": container with ID starting with 917832e314d8098db957b8fcb6b0b33c859932d72385434edfbbfb71896a658b not found: ID does not exist" Nov 03 06:11:22.102550 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.102482 4855 scope.go:122] "RemoveContainer" containerID="4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218" Nov 03 06:11:22.103223 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:22.102902 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218\": container with ID starting with 4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218 not found: ID does not exist" containerID="4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218" Nov 03 06:11:22.103223 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.102930 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218"} err="failed to get container status \"4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218\": rpc error: code = NotFound desc = could not find container \"4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218\": container with ID starting with 4aa83d47f95a85abcaf4453c116b937164d21c72c197aff12c64a4a65d609218 not found: ID does not exist" Nov 03 06:11:22.103223 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.102951 4855 scope.go:122] "RemoveContainer" containerID="fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8" Nov 03 06:11:22.126698 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.126183 4855 scope.go:122] "RemoveContainer" containerID="fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8" Nov 03 06:11:22.126698 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:22.126537 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8\": container with ID starting with fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8 not found: ID does not exist" containerID="fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8" Nov 03 06:11:22.126698 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.126570 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8"} err="failed to get container status \"fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8\": rpc error: code = NotFound desc = could not find container \"fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8\": container with ID starting with fd26fa12948e3eeebb48d08a40f49fa8841c11050a1122a57de557fad8aaf9e8 not found: ID does not exist" Nov 03 06:11:22.140618 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.140598 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:11:22.143085 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.143040 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:11:22.146098 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.146074 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t"] Nov 03 06:11:22.150494 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.150475 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-55p5t"] Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.174531 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-socket\") pod \"8f7ea471-1f77-49c0-92dc-139f56db1d14\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.174598 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-credential-socket\") pod \"82859956-f5dd-43be-894b-2e78907f7505\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.174631 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-credential-socket\") pod \"8f7ea471-1f77-49c0-92dc-139f56db1d14\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.174673 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-socket\") pod \"82859956-f5dd-43be-894b-2e78907f7505\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.174720 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-podinfo\") pod \"8f7ea471-1f77-49c0-92dc-139f56db1d14\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.174753 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-kube-api-access-8r7pb\" (UniqueName: \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-kube-api-access-8r7pb\") pod \"82859956-f5dd-43be-894b-2e78907f7505\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.174819 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-token\" (UniqueName: \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-token\") pod \"8f7ea471-1f77-49c0-92dc-139f56db1d14\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.174852 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7ea471-1f77-49c0-92dc-139f56db1d14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7ea471-1f77-49c0-92dc-139f56db1d14-istiod-ca-cert\") pod \"8f7ea471-1f77-49c0-92dc-139f56db1d14\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.174920 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-kube-api-access-pp5jc\" (UniqueName: \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-kube-api-access-pp5jc\") pod \"8f7ea471-1f77-49c0-92dc-139f56db1d14\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.174959 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82859956-f5dd-43be-894b-2e78907f7505-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82859956-f5dd-43be-894b-2e78907f7505-istio-podinfo\") pod \"82859956-f5dd-43be-894b-2e78907f7505\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.175027 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-data\") pod \"82859956-f5dd-43be-894b-2e78907f7505\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.175063 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-envoy\") pod \"82859956-f5dd-43be-894b-2e78907f7505\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.175123 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-certs\") pod \"82859956-f5dd-43be-894b-2e78907f7505\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.175151 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-istio-token\" (UniqueName: \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-istio-token\") pod \"82859956-f5dd-43be-894b-2e78907f7505\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.175209 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-certs\") pod \"8f7ea471-1f77-49c0-92dc-139f56db1d14\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.175262 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-data\") pod \"8f7ea471-1f77-49c0-92dc-139f56db1d14\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.175334 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82859956-f5dd-43be-894b-2e78907f7505-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82859956-f5dd-43be-894b-2e78907f7505-istiod-ca-cert\") pod \"82859956-f5dd-43be-894b-2e78907f7505\" (UID: \"82859956-f5dd-43be-894b-2e78907f7505\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.175361 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-envoy\") pod \"8f7ea471-1f77-49c0-92dc-139f56db1d14\" (UID: \"8f7ea471-1f77-49c0-92dc-139f56db1d14\") " Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.177111 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.177139 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmsjx\" (UniqueName: \"kubernetes.io/projected/ec3a0b0e-10e5-4605-aa0e-67745585646d-kube-api-access-mmsjx\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.177154 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.177167 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.177180 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.177194 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.177207 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.177220 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3a0b0e-10e5-4605-aa0e-67745585646d-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.177417 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.177233 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3a0b0e-10e5-4605-aa0e-67745585646d-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.181757 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.181713 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-data" pod "82859956-f5dd-43be-894b-2e78907f7505" (UID: "82859956-f5dd-43be-894b-2e78907f7505"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.182016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.181994 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-socket" pod "82859956-f5dd-43be-894b-2e78907f7505" (UID: "82859956-f5dd-43be-894b-2e78907f7505"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.182202 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.182185 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-socket" pod "8f7ea471-1f77-49c0-92dc-139f56db1d14" (UID: "8f7ea471-1f77-49c0-92dc-139f56db1d14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.182323 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.182309 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-credential-socket" pod "82859956-f5dd-43be-894b-2e78907f7505" (UID: "82859956-f5dd-43be-894b-2e78907f7505"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.182475 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.182456 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-credential-socket" pod "8f7ea471-1f77-49c0-92dc-139f56db1d14" (UID: "8f7ea471-1f77-49c0-92dc-139f56db1d14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.182547 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.182499 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-certs" pod "82859956-f5dd-43be-894b-2e78907f7505" (UID: "82859956-f5dd-43be-894b-2e78907f7505"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.182749 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.182650 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-certs" pod "8f7ea471-1f77-49c0-92dc-139f56db1d14" (UID: "8f7ea471-1f77-49c0-92dc-139f56db1d14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.183087 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.182862 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-data" pod "8f7ea471-1f77-49c0-92dc-139f56db1d14" (UID: "8f7ea471-1f77-49c0-92dc-139f56db1d14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.183902 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.183526 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-envoy" pod "8f7ea471-1f77-49c0-92dc-139f56db1d14" (UID: "8f7ea471-1f77-49c0-92dc-139f56db1d14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.183902 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.183828 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82859956-f5dd-43be-894b-2e78907f7505-istio-podinfo" pod "82859956-f5dd-43be-894b-2e78907f7505" (UID: "82859956-f5dd-43be-894b-2e78907f7505"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:22.183902 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.183854 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7ea471-1f77-49c0-92dc-139f56db1d14-istiod-ca-cert" pod "8f7ea471-1f77-49c0-92dc-139f56db1d14" (UID: "8f7ea471-1f77-49c0-92dc-139f56db1d14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:22.184601 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.184556 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-kube-api-access-pp5jc" pod "8f7ea471-1f77-49c0-92dc-139f56db1d14" (UID: "8f7ea471-1f77-49c0-92dc-139f56db1d14"). InnerVolumeSpecName "kube-api-access-pp5jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:22.185951 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.185918 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82859956-f5dd-43be-894b-2e78907f7505-istiod-ca-cert" pod "82859956-f5dd-43be-894b-2e78907f7505" (UID: "82859956-f5dd-43be-894b-2e78907f7505"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:22.187533 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.187483 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-podinfo" pod "8f7ea471-1f77-49c0-92dc-139f56db1d14" (UID: "8f7ea471-1f77-49c0-92dc-139f56db1d14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:22.187622 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.187554 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-token" pod "8f7ea471-1f77-49c0-92dc-139f56db1d14" (UID: "8f7ea471-1f77-49c0-92dc-139f56db1d14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:22.187809 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.187784 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-istio-token" pod "82859956-f5dd-43be-894b-2e78907f7505" (UID: "82859956-f5dd-43be-894b-2e78907f7505"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:22.187909 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.187806 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-kube-api-access-8r7pb" pod "82859956-f5dd-43be-894b-2e78907f7505" (UID: "82859956-f5dd-43be-894b-2e78907f7505"). InnerVolumeSpecName "kube-api-access-8r7pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:22.188097 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.188077 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-envoy" pod "82859956-f5dd-43be-894b-2e78907f7505" (UID: "82859956-f5dd-43be-894b-2e78907f7505"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.193129 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.193074 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd"] Nov 03 06:11:22.196320 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.196301 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-5gvjd"] Nov 03 06:11:22.273745 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.273711 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s"] Nov 03 06:11:22.275688 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.275662 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-fj49s"] Nov 03 06:11:22.278698 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278677 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5jc\" (UniqueName: \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-kube-api-access-pp5jc\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278698 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278698 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82859956-f5dd-43be-894b-2e78907f7505-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278709 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278719 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278727 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278736 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278744 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278752 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278760 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82859956-f5dd-43be-894b-2e78907f7505-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278767 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278775 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278782 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278790 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ea471-1f77-49c0-92dc-139f56db1d14-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278799 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82859956-f5dd-43be-894b-2e78907f7505-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278807 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278816 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r7pb\" (UniqueName: \"kubernetes.io/projected/82859956-f5dd-43be-894b-2e78907f7505-kube-api-access-8r7pb\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278823 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7ea471-1f77-49c0-92dc-139f56db1d14-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.278826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.278832 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7ea471-1f77-49c0-92dc-139f56db1d14-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.338069 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.337304 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" path="/var/lib/kubelet/pods/21990fba-a699-4ba0-8a0d-ac9025d3c129/volumes" Nov 03 06:11:22.338069 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.337975 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" path="/var/lib/kubelet/pods/8e65c31d-5628-4811-a84e-405aa9eb9188/volumes" Nov 03 06:11:22.338585 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.338564 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" path="/var/lib/kubelet/pods/cafef133-515b-4fdf-a799-cf10a42224f8/volumes" Nov 03 06:11:22.339337 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.339318 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" path="/var/lib/kubelet/pods/d48278d0-aa0f-4f79-af81-39f95b6c5495/volumes" Nov 03 06:11:22.339779 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.339761 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" path="/var/lib/kubelet/pods/d926084c-dafb-4fd7-adc3-928dc9036a92/volumes" Nov 03 06:11:22.340231 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.340212 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" path="/var/lib/kubelet/pods/ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62/volumes" Nov 03 06:11:22.341227 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.341201 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee92591f-8473-4981-8af8-bb60f096c653" path="/var/lib/kubelet/pods/ee92591f-8473-4981-8af8-bb60f096c653/volumes" Nov 03 06:11:22.466787 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.466757 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:11:22.480916 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.480885 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs"] Nov 03 06:11:22.482538 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.482509 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-credential-socket\") pod \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " Nov 03 06:11:22.482720 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.482614 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istiod-ca-cert\") pod \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " Nov 03 06:11:22.482720 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.482695 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-podinfo\") pod \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " Nov 03 06:11:22.483176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.482740 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-token\" (UniqueName: \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-token\") pod \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " Nov 03 06:11:22.483176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.482792 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-envoy\") pod \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " Nov 03 06:11:22.483176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.482796 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-credential-socket" pod "6b19d23e-3efa-4c72-a703-6610bd96a3c4" (UID: "6b19d23e-3efa-4c72-a703-6610bd96a3c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.483176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.482867 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-data\") pod \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " Nov 03 06:11:22.483176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.482963 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-socket\") pod \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " Nov 03 06:11:22.483176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.483005 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-kube-api-access-cfsjn\" (UniqueName: \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-kube-api-access-cfsjn\") pod \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " Nov 03 06:11:22.483176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.483092 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-certs\") pod \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\" (UID: \"6b19d23e-3efa-4c72-a703-6610bd96a3c4\") " Nov 03 06:11:22.483665 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.483644 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.483719 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.483696 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istiod-ca-cert" pod "6b19d23e-3efa-4c72-a703-6610bd96a3c4" (UID: "6b19d23e-3efa-4c72-a703-6610bd96a3c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:22.483887 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.483865 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-certs" pod "6b19d23e-3efa-4c72-a703-6610bd96a3c4" (UID: "6b19d23e-3efa-4c72-a703-6610bd96a3c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.483973 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.483954 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-socket" pod "6b19d23e-3efa-4c72-a703-6610bd96a3c4" (UID: "6b19d23e-3efa-4c72-a703-6610bd96a3c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.484114 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.484100 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-data" pod "6b19d23e-3efa-4c72-a703-6610bd96a3c4" (UID: "6b19d23e-3efa-4c72-a703-6610bd96a3c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.485866 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.485641 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-envoy" pod "6b19d23e-3efa-4c72-a703-6610bd96a3c4" (UID: "6b19d23e-3efa-4c72-a703-6610bd96a3c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:22.485866 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.485787 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-27pcs"] Nov 03 06:11:22.486272 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.486251 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-podinfo" pod "6b19d23e-3efa-4c72-a703-6610bd96a3c4" (UID: "6b19d23e-3efa-4c72-a703-6610bd96a3c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:22.486340 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.486285 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-kube-api-access-cfsjn" pod "6b19d23e-3efa-4c72-a703-6610bd96a3c4" (UID: "6b19d23e-3efa-4c72-a703-6610bd96a3c4"). InnerVolumeSpecName "kube-api-access-cfsjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:22.486457 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.486440 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-token" pod "6b19d23e-3efa-4c72-a703-6610bd96a3c4" (UID: "6b19d23e-3efa-4c72-a703-6610bd96a3c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:22.585207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.585174 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.585207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.585206 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.585207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.585217 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.585567 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.585229 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.585567 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.585240 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.585567 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.585251 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.585567 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.585262 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfsjn\" (UniqueName: \"kubernetes.io/projected/6b19d23e-3efa-4c72-a703-6610bd96a3c4-kube-api-access-cfsjn\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:22.585567 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:22.585275 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b19d23e-3efa-4c72-a703-6610bd96a3c4-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:23.009566 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.009489 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" Nov 03 06:11:23.009708 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.009488 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn" event={"ID":"82859956-f5dd-43be-894b-2e78907f7505","Type":"ContainerDied","Data":"7462b8c307b88cb0e91062e72ade10b7d8fcb81254a12d475cfabe6a0ee8f921"} Nov 03 06:11:23.009708 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.009647 4855 scope.go:122] "RemoveContainer" containerID="c099a8b6c465176dc88af56014616d5c8cd601520b6d2832db31e643e19861ad" Nov 03 06:11:23.011104 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.011043 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" event={"ID":"ec3a0b0e-10e5-4605-aa0e-67745585646d","Type":"ContainerDied","Data":"5af2190cbfb5c7e4afe8b241c72657f105c1653a7ac4d08f7a3ca178ccbcadb9"} Nov 03 06:11:23.011104 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.011061 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb" Nov 03 06:11:23.017525 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.017506 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" Nov 03 06:11:23.017525 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.017521 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2" event={"ID":"6b19d23e-3efa-4c72-a703-6610bd96a3c4","Type":"ContainerDied","Data":"4165a5497f1321cb365c1dd85d348761ddab106b03490322b4eaced50809a526"} Nov 03 06:11:23.018881 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.018851 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" event={"ID":"8f7ea471-1f77-49c0-92dc-139f56db1d14","Type":"ContainerDied","Data":"8b2fabc501cddae8de27712cc7982ca1f4d7c67ac0fa44c5a29a71824a70386f"} Nov 03 06:11:23.018977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.018858 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw" Nov 03 06:11:23.020297 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.020274 4855 scope.go:122] "RemoveContainer" containerID="8c9ac4e3dd1e867cc099dbdd318db6729f0fd37ee3058499c513675eeeb5e878" Nov 03 06:11:23.029151 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.029135 4855 scope.go:122] "RemoveContainer" containerID="eb13878edd20bc24955729046363a55f7c55d6fd565180eba2f3df9678a623ab" Nov 03 06:11:23.037743 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.037726 4855 scope.go:122] "RemoveContainer" containerID="55491e7918d0c8346bfafcff110bb85b41026bbe975dcbc5bd01054ea3a5939f" Nov 03 06:11:23.073124 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.073094 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn"] Nov 03 06:11:23.081456 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.081432 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-sbthn"] Nov 03 06:11:23.132334 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.132303 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb"] Nov 03 06:11:23.134631 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.134612 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-rcprb"] Nov 03 06:11:23.177301 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.177270 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw"] Nov 03 06:11:23.182312 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.182289 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-ccpqw"] Nov 03 06:11:23.238217 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.238181 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2"] Nov 03 06:11:23.245075 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:23.245049 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-g95q2"] Nov 03 06:11:24.332938 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:24.332906 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" path="/var/lib/kubelet/pods/6b19d23e-3efa-4c72-a703-6610bd96a3c4/volumes" Nov 03 06:11:24.333432 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:24.333416 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" path="/var/lib/kubelet/pods/6c5c4df2-55ce-4bd3-b10d-10ba796dce4c/volumes" Nov 03 06:11:24.333866 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:24.333852 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82859956-f5dd-43be-894b-2e78907f7505" path="/var/lib/kubelet/pods/82859956-f5dd-43be-894b-2e78907f7505/volumes" Nov 03 06:11:24.334654 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:24.334639 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" path="/var/lib/kubelet/pods/8f7ea471-1f77-49c0-92dc-139f56db1d14/volumes" Nov 03 06:11:24.335087 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:24.335073 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" path="/var/lib/kubelet/pods/ec3a0b0e-10e5-4605-aa0e-67745585646d/volumes" Nov 03 06:11:24.801284 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:24.801242 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" containerID="cri-o://0240ac82bbf82e1998b19860048bc821dabb1eaf7d45e60c017e64ca68c3ccb2" gracePeriod=45 Nov 03 06:11:25.039539 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.039506 4855 generic.go:356] "Generic (PLEG): container finished" podID="a6892c41-6866-43ee-82da-594bc4e28b29" containerID="0240ac82bbf82e1998b19860048bc821dabb1eaf7d45e60c017e64ca68c3ccb2" exitCode=0 Nov 03 06:11:25.039653 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.039583 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" event={"ID":"a6892c41-6866-43ee-82da-594bc4e28b29","Type":"ContainerDied","Data":"0240ac82bbf82e1998b19860048bc821dabb1eaf7d45e60c017e64ca68c3ccb2"} Nov 03 06:11:25.061237 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.061183 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:11:25.111808 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.111776 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-model-cache\") pod \"a6892c41-6866-43ee-82da-594bc4e28b29\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " Nov 03 06:11:25.111986 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.111845 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6892c41-6866-43ee-82da-594bc4e28b29-kube-api-access-dg8z6\" (UniqueName: \"kubernetes.io/projected/a6892c41-6866-43ee-82da-594bc4e28b29-kube-api-access-dg8z6\") pod \"a6892c41-6866-43ee-82da-594bc4e28b29\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " Nov 03 06:11:25.111986 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.111880 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-dshm\") pod \"a6892c41-6866-43ee-82da-594bc4e28b29\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " Nov 03 06:11:25.111986 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.111917 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-tmp-dir\") pod \"a6892c41-6866-43ee-82da-594bc4e28b29\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " Nov 03 06:11:25.111986 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.111945 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-home\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-home\") pod \"a6892c41-6866-43ee-82da-594bc4e28b29\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " Nov 03 06:11:25.112219 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.112010 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6892c41-6866-43ee-82da-594bc4e28b29-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6892c41-6866-43ee-82da-594bc4e28b29-tls-certs\") pod \"a6892c41-6866-43ee-82da-594bc4e28b29\" (UID: \"a6892c41-6866-43ee-82da-594bc4e28b29\") " Nov 03 06:11:25.113088 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.113026 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-home" pod "a6892c41-6866-43ee-82da-594bc4e28b29" (UID: "a6892c41-6866-43ee-82da-594bc4e28b29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:25.113316 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.113205 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-tmp-dir" pod "a6892c41-6866-43ee-82da-594bc4e28b29" (UID: "a6892c41-6866-43ee-82da-594bc4e28b29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:25.114731 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.114702 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-model-cache" pod "a6892c41-6866-43ee-82da-594bc4e28b29" (UID: "a6892c41-6866-43ee-82da-594bc4e28b29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:25.115548 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.115261 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-dshm" pod "a6892c41-6866-43ee-82da-594bc4e28b29" (UID: "a6892c41-6866-43ee-82da-594bc4e28b29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:25.121814 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.121782 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6892c41-6866-43ee-82da-594bc4e28b29-tls-certs" pod "a6892c41-6866-43ee-82da-594bc4e28b29" (UID: "a6892c41-6866-43ee-82da-594bc4e28b29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:11:25.121894 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.121786 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6892c41-6866-43ee-82da-594bc4e28b29-kube-api-access-dg8z6" pod "a6892c41-6866-43ee-82da-594bc4e28b29" (UID: "a6892c41-6866-43ee-82da-594bc4e28b29"). InnerVolumeSpecName "kube-api-access-dg8z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:25.214422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.214374 4855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-model-cache\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:25.214422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.214426 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg8z6\" (UniqueName: \"kubernetes.io/projected/a6892c41-6866-43ee-82da-594bc4e28b29-kube-api-access-dg8z6\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:25.215001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.214442 4855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-dshm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:25.215001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.214456 4855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-tmp-dir\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:25.215001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.214467 4855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6892c41-6866-43ee-82da-594bc4e28b29-home\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:25.215001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.214479 4855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6892c41-6866-43ee-82da-594bc4e28b29-tls-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:25.241153 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.241116 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5"] Nov 03 06:11:25.241422 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.241375 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" podUID="34d1643a-db61-492d-8b12-ef21e5eb0834" containerName="maas-api" containerID="cri-o://11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc" gracePeriod=30 Nov 03 06:11:25.563421 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.559535 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:11:25.632813 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.632734 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-metrics-certs\") pod \"34d1643a-db61-492d-8b12-ef21e5eb0834\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " Nov 03 06:11:25.632813 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.632792 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d1643a-db61-492d-8b12-ef21e5eb0834-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d1643a-db61-492d-8b12-ef21e5eb0834-openshift-service-ca\") pod \"34d1643a-db61-492d-8b12-ef21e5eb0834\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " Nov 03 06:11:25.633044 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.632890 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d1643a-db61-492d-8b12-ef21e5eb0834-kube-api-access-vmxsd\" (UniqueName: \"kubernetes.io/projected/34d1643a-db61-492d-8b12-ef21e5eb0834-kube-api-access-vmxsd\") pod \"34d1643a-db61-492d-8b12-ef21e5eb0834\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " Nov 03 06:11:25.633044 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.632960 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-maas-api-tls\") pod \"34d1643a-db61-492d-8b12-ef21e5eb0834\" (UID: \"34d1643a-db61-492d-8b12-ef21e5eb0834\") " Nov 03 06:11:25.633203 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.633168 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d1643a-db61-492d-8b12-ef21e5eb0834-openshift-service-ca" pod "34d1643a-db61-492d-8b12-ef21e5eb0834" (UID: "34d1643a-db61-492d-8b12-ef21e5eb0834"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:25.633537 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.633507 4855 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d1643a-db61-492d-8b12-ef21e5eb0834-openshift-service-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:25.635098 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.635076 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-maas-api-tls" pod "34d1643a-db61-492d-8b12-ef21e5eb0834" (UID: "34d1643a-db61-492d-8b12-ef21e5eb0834"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:11:25.635379 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.635362 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-metrics-certs" pod "34d1643a-db61-492d-8b12-ef21e5eb0834" (UID: "34d1643a-db61-492d-8b12-ef21e5eb0834"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:11:25.635468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.635433 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d1643a-db61-492d-8b12-ef21e5eb0834-kube-api-access-vmxsd" pod "34d1643a-db61-492d-8b12-ef21e5eb0834" (UID: "34d1643a-db61-492d-8b12-ef21e5eb0834"). InnerVolumeSpecName "kube-api-access-vmxsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:25.771994 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.758642 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:25.772286 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.772260 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmxsd\" (UniqueName: \"kubernetes.io/projected/34d1643a-db61-492d-8b12-ef21e5eb0834-kube-api-access-vmxsd\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:25.772468 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:25.772448 4855 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d1643a-db61-492d-8b12-ef21e5eb0834-maas-api-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:26.046248 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.046221 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" Nov 03 06:11:26.046414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.046220 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj" event={"ID":"a6892c41-6866-43ee-82da-594bc4e28b29","Type":"ContainerDied","Data":"7c88f1ca840b354763b7bf7ba65ff624c88afd0937aeddbfb2044ef959390038"} Nov 03 06:11:26.046489 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.046418 4855 scope.go:122] "RemoveContainer" containerID="0240ac82bbf82e1998b19860048bc821dabb1eaf7d45e60c017e64ca68c3ccb2" Nov 03 06:11:26.047884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.047573 4855 generic.go:356] "Generic (PLEG): container finished" podID="34d1643a-db61-492d-8b12-ef21e5eb0834" containerID="11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc" exitCode=0 Nov 03 06:11:26.047884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.047632 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" Nov 03 06:11:26.047884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.047648 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" event={"ID":"34d1643a-db61-492d-8b12-ef21e5eb0834","Type":"ContainerDied","Data":"11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc"} Nov 03 06:11:26.047884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.047707 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5" event={"ID":"34d1643a-db61-492d-8b12-ef21e5eb0834","Type":"ContainerDied","Data":"f9ea53686d3aa37f26a4a3f227fc5f6f6c729dc3eef9935444d0a9487e848d13"} Nov 03 06:11:26.055828 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.055774 4855 scope.go:122] "RemoveContainer" containerID="11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc" Nov 03 06:11:26.064184 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.064165 4855 scope.go:122] "RemoveContainer" containerID="11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc" Nov 03 06:11:26.064488 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:26.064469 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc\": container with ID starting with 11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc not found: ID does not exist" containerID="11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc" Nov 03 06:11:26.064549 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.064493 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc"} err="failed to get container status \"11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc\": rpc error: code = NotFound desc = could not find container \"11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc\": container with ID starting with 11d0dae7ab64bea1ec7d388c13c77a2de70d5e312fc489770cae334024189fdc not found: ID does not exist" Nov 03 06:11:26.117658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.117626 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5"] Nov 03 06:11:26.130542 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.130513 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5c3542-d69b78b65-mg7g5"] Nov 03 06:11:26.174509 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.174482 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj"] Nov 03 06:11:26.204309 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.204279 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-w1-9688d8-kserve-7599d77959-jrjvj"] Nov 03 06:11:26.333294 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.333215 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d1643a-db61-492d-8b12-ef21e5eb0834" path="/var/lib/kubelet/pods/34d1643a-db61-492d-8b12-ef21e5eb0834/volumes" Nov 03 06:11:26.333699 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:26.333683 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" path="/var/lib/kubelet/pods/a6892c41-6866-43ee-82da-594bc4e28b29/volumes" Nov 03 06:11:28.883724 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:28.883690 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:11:32.844780 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:32.844746 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb"] Nov 03 06:11:32.845224 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:32.844932 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" containerID="cri-o://7794b81a70c72b7b909cf1adeb6035a3765690ea4c1c4c9287e1d02b472955cc" gracePeriod=30 Nov 03 06:11:32.861698 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:32.858588 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk"] Nov 03 06:11:32.861698 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:32.858878 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerName="istio-proxy" containerID="cri-o://7e0e80823f4ba203a79b6527166f92c61ec0449913fb8cea2de4587069cc6abc" gracePeriod=30 Nov 03 06:11:32.871589 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:32.871562 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll"] Nov 03 06:11:32.871764 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:32.871728 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" containerID="cri-o://16dd80b96b3f0e0b29eedb81d1b6667865540f07bc45c8a0d92016583bd07747" gracePeriod=30 Nov 03 06:11:32.879244 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:32.879222 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n"] Nov 03 06:11:32.879461 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:32.879440 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" containerID="cri-o://7d20f21835eee29c53b5ffda686806674f84e6264eb08f88bbb34c6a7fdd7a62" gracePeriod=30 Nov 03 06:11:34.779921 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:34.779883 4855 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:34.780323 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:34.779942 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:38.130204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.130172 4855 generic.go:356] "Generic (PLEG): container finished" podID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerID="7794b81a70c72b7b909cf1adeb6035a3765690ea4c1c4c9287e1d02b472955cc" exitCode=0 Nov 03 06:11:38.130640 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.130254 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" event={"ID":"87cee8e8-09ff-49e8-89ee-04c82a0f50a9","Type":"ContainerDied","Data":"7794b81a70c72b7b909cf1adeb6035a3765690ea4c1c4c9287e1d02b472955cc"} Nov 03 06:11:38.132181 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.132160 4855 generic.go:356] "Generic (PLEG): container finished" podID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerID="7e0e80823f4ba203a79b6527166f92c61ec0449913fb8cea2de4587069cc6abc" exitCode=0 Nov 03 06:11:38.132288 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.132231 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" event={"ID":"c2a24180-c194-4ead-810e-5e20c52cfbb9","Type":"ContainerDied","Data":"7e0e80823f4ba203a79b6527166f92c61ec0449913fb8cea2de4587069cc6abc"} Nov 03 06:11:38.134035 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.134018 4855 generic.go:356] "Generic (PLEG): container finished" podID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerID="16dd80b96b3f0e0b29eedb81d1b6667865540f07bc45c8a0d92016583bd07747" exitCode=0 Nov 03 06:11:38.134119 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.134083 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" event={"ID":"6a908b4b-a058-4eea-9d4b-05899552a98c","Type":"ContainerDied","Data":"16dd80b96b3f0e0b29eedb81d1b6667865540f07bc45c8a0d92016583bd07747"} Nov 03 06:11:38.135803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.135783 4855 generic.go:356] "Generic (PLEG): container finished" podID="43948c14-233c-4679-93bc-b73c2f09725f" containerID="7d20f21835eee29c53b5ffda686806674f84e6264eb08f88bbb34c6a7fdd7a62" exitCode=0 Nov 03 06:11:38.135881 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.135819 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" event={"ID":"43948c14-233c-4679-93bc-b73c2f09725f","Type":"ContainerDied","Data":"7d20f21835eee29c53b5ffda686806674f84e6264eb08f88bbb34c6a7fdd7a62"} Nov 03 06:11:38.149420 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.149376 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:11:38.229520 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.229496 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:11:38.234555 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.234534 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:11:38.238926 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.238899 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:11:38.246755 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.246729 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-certs\") pod \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " Nov 03 06:11:38.246863 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.246791 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-podinfo\") pod \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " Nov 03 06:11:38.246929 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.246885 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-data\") pod \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " Nov 03 06:11:38.246990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.246931 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istiod-ca-cert\") pod \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " Nov 03 06:11:38.246990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.246958 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-certs" pod "87cee8e8-09ff-49e8-89ee-04c82a0f50a9" (UID: "87cee8e8-09ff-49e8-89ee-04c82a0f50a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.246990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.246973 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-credential-socket\") pod \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " Nov 03 06:11:38.247130 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.247110 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-token\" (UniqueName: \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-token\") pod \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " Nov 03 06:11:38.247181 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.247136 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-data" pod "87cee8e8-09ff-49e8-89ee-04c82a0f50a9" (UID: "87cee8e8-09ff-49e8-89ee-04c82a0f50a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.247181 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.247154 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-socket\") pod \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " Nov 03 06:11:38.247285 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.247183 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-credential-socket" pod "87cee8e8-09ff-49e8-89ee-04c82a0f50a9" (UID: "87cee8e8-09ff-49e8-89ee-04c82a0f50a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.247285 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.247187 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-envoy\") pod \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " Nov 03 06:11:38.247285 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.247238 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-kube-api-access-7drqd\" (UniqueName: \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-kube-api-access-7drqd\") pod \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\" (UID: \"87cee8e8-09ff-49e8-89ee-04c82a0f50a9\") " Nov 03 06:11:38.247729 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.247524 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istiod-ca-cert" pod "87cee8e8-09ff-49e8-89ee-04c82a0f50a9" (UID: "87cee8e8-09ff-49e8-89ee-04c82a0f50a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:38.248038 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.248009 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-socket" pod "87cee8e8-09ff-49e8-89ee-04c82a0f50a9" (UID: "87cee8e8-09ff-49e8-89ee-04c82a0f50a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.248735 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.248709 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.248818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.248741 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.248818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.248756 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.248818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.248770 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.248818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.248784 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.250643 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.250608 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-token" pod "87cee8e8-09ff-49e8-89ee-04c82a0f50a9" (UID: "87cee8e8-09ff-49e8-89ee-04c82a0f50a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:38.252263 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.252243 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-envoy" pod "87cee8e8-09ff-49e8-89ee-04c82a0f50a9" (UID: "87cee8e8-09ff-49e8-89ee-04c82a0f50a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.252433 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.252385 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-kube-api-access-7drqd" pod "87cee8e8-09ff-49e8-89ee-04c82a0f50a9" (UID: "87cee8e8-09ff-49e8-89ee-04c82a0f50a9"). InnerVolumeSpecName "kube-api-access-7drqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:38.253058 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.253031 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-podinfo" pod "87cee8e8-09ff-49e8-89ee-04c82a0f50a9" (UID: "87cee8e8-09ff-49e8-89ee-04c82a0f50a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:38.349352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349319 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-istio-token\" (UniqueName: \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-istio-token\") pod \"43948c14-233c-4679-93bc-b73c2f09725f\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " Nov 03 06:11:38.349511 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349377 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-certs\") pod \"43948c14-233c-4679-93bc-b73c2f09725f\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " Nov 03 06:11:38.349511 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349464 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43948c14-233c-4679-93bc-b73c2f09725f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43948c14-233c-4679-93bc-b73c2f09725f-istiod-ca-cert\") pod \"43948c14-233c-4679-93bc-b73c2f09725f\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " Nov 03 06:11:38.349511 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349505 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-certs\") pod \"6a908b4b-a058-4eea-9d4b-05899552a98c\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " Nov 03 06:11:38.349669 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349540 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-credential-socket\") pod \"c2a24180-c194-4ead-810e-5e20c52cfbb9\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " Nov 03 06:11:38.349669 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349571 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-certs\") pod \"c2a24180-c194-4ead-810e-5e20c52cfbb9\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " Nov 03 06:11:38.349669 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349600 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-envoy\") pod \"c2a24180-c194-4ead-810e-5e20c52cfbb9\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " Nov 03 06:11:38.349669 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349630 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-socket\") pod \"43948c14-233c-4679-93bc-b73c2f09725f\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " Nov 03 06:11:38.349669 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349659 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-socket\") pod \"c2a24180-c194-4ead-810e-5e20c52cfbb9\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " Nov 03 06:11:38.349889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349715 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43948c14-233c-4679-93bc-b73c2f09725f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43948c14-233c-4679-93bc-b73c2f09725f-istio-podinfo\") pod \"43948c14-233c-4679-93bc-b73c2f09725f\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " Nov 03 06:11:38.349889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349750 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-certs" pod "43948c14-233c-4679-93bc-b73c2f09725f" (UID: "43948c14-233c-4679-93bc-b73c2f09725f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.349889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349765 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-podinfo\") pod \"6a908b4b-a058-4eea-9d4b-05899552a98c\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " Nov 03 06:11:38.349889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349830 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-envoy\") pod \"6a908b4b-a058-4eea-9d4b-05899552a98c\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " Nov 03 06:11:38.349889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349841 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43948c14-233c-4679-93bc-b73c2f09725f-istiod-ca-cert" pod "43948c14-233c-4679-93bc-b73c2f09725f" (UID: "43948c14-233c-4679-93bc-b73c2f09725f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:38.349889 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349878 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-credential-socket\") pod \"43948c14-233c-4679-93bc-b73c2f09725f\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " Nov 03 06:11:38.350155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349917 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-socket\") pod \"6a908b4b-a058-4eea-9d4b-05899552a98c\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " Nov 03 06:11:38.350155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349947 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-token\" (UniqueName: \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-token\") pod \"c2a24180-c194-4ead-810e-5e20c52cfbb9\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " Nov 03 06:11:38.350155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.349989 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a908b4b-a058-4eea-9d4b-05899552a98c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a908b4b-a058-4eea-9d4b-05899552a98c-istiod-ca-cert\") pod \"6a908b4b-a058-4eea-9d4b-05899552a98c\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " Nov 03 06:11:38.350155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350033 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-envoy\") pod \"43948c14-233c-4679-93bc-b73c2f09725f\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " Nov 03 06:11:38.350155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350062 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-token\" (UniqueName: \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-token\") pod \"6a908b4b-a058-4eea-9d4b-05899552a98c\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " Nov 03 06:11:38.350155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350111 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-kube-api-access-t6pw5\" (UniqueName: \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-kube-api-access-t6pw5\") pod \"43948c14-233c-4679-93bc-b73c2f09725f\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " Nov 03 06:11:38.350155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350149 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-kube-api-access-945ph\" (UniqueName: \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-kube-api-access-945ph\") pod \"c2a24180-c194-4ead-810e-5e20c52cfbb9\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " Nov 03 06:11:38.350486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350183 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-credential-socket\") pod \"6a908b4b-a058-4eea-9d4b-05899552a98c\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " Nov 03 06:11:38.350486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350230 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a24180-c194-4ead-810e-5e20c52cfbb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a24180-c194-4ead-810e-5e20c52cfbb9-istiod-ca-cert\") pod \"c2a24180-c194-4ead-810e-5e20c52cfbb9\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " Nov 03 06:11:38.350486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350259 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-kube-api-access-66zbt\" (UniqueName: \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-kube-api-access-66zbt\") pod \"6a908b4b-a058-4eea-9d4b-05899552a98c\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " Nov 03 06:11:38.350486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350289 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-podinfo\") pod \"c2a24180-c194-4ead-810e-5e20c52cfbb9\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " Nov 03 06:11:38.350486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350338 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-data\") pod \"c2a24180-c194-4ead-810e-5e20c52cfbb9\" (UID: \"c2a24180-c194-4ead-810e-5e20c52cfbb9\") " Nov 03 06:11:38.350486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350368 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-data\") pod \"6a908b4b-a058-4eea-9d4b-05899552a98c\" (UID: \"6a908b4b-a058-4eea-9d4b-05899552a98c\") " Nov 03 06:11:38.350486 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350449 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-data\") pod \"43948c14-233c-4679-93bc-b73c2f09725f\" (UID: \"43948c14-233c-4679-93bc-b73c2f09725f\") " Nov 03 06:11:38.350913 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350892 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.350968 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350921 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.350968 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350936 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43948c14-233c-4679-93bc-b73c2f09725f-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.350968 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350949 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.350968 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350962 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.351136 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.350975 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7drqd\" (UniqueName: \"kubernetes.io/projected/87cee8e8-09ff-49e8-89ee-04c82a0f50a9-kube-api-access-7drqd\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.351238 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.351217 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-data" pod "43948c14-233c-4679-93bc-b73c2f09725f" (UID: "43948c14-233c-4679-93bc-b73c2f09725f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.351679 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.351651 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-credential-socket" pod "c2a24180-c194-4ead-810e-5e20c52cfbb9" (UID: "c2a24180-c194-4ead-810e-5e20c52cfbb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.351925 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.351904 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-certs" pod "6a908b4b-a058-4eea-9d4b-05899552a98c" (UID: "6a908b4b-a058-4eea-9d4b-05899552a98c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.352142 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.352115 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-istio-token" pod "43948c14-233c-4679-93bc-b73c2f09725f" (UID: "43948c14-233c-4679-93bc-b73c2f09725f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:38.352278 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.352253 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-podinfo" pod "6a908b4b-a058-4eea-9d4b-05899552a98c" (UID: "6a908b4b-a058-4eea-9d4b-05899552a98c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:38.352368 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.352345 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-socket" pod "c2a24180-c194-4ead-810e-5e20c52cfbb9" (UID: "c2a24180-c194-4ead-810e-5e20c52cfbb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.352614 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.352591 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-socket" pod "43948c14-233c-4679-93bc-b73c2f09725f" (UID: "43948c14-233c-4679-93bc-b73c2f09725f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.352682 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.352661 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-certs" pod "c2a24180-c194-4ead-810e-5e20c52cfbb9" (UID: "c2a24180-c194-4ead-810e-5e20c52cfbb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.352828 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.352806 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-envoy" pod "c2a24180-c194-4ead-810e-5e20c52cfbb9" (UID: "c2a24180-c194-4ead-810e-5e20c52cfbb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.353970 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.353842 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-envoy" pod "6a908b4b-a058-4eea-9d4b-05899552a98c" (UID: "6a908b4b-a058-4eea-9d4b-05899552a98c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.354274 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.354061 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-credential-socket" pod "6a908b4b-a058-4eea-9d4b-05899552a98c" (UID: "6a908b4b-a058-4eea-9d4b-05899552a98c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.354274 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.354159 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-credential-socket" pod "43948c14-233c-4679-93bc-b73c2f09725f" (UID: "43948c14-233c-4679-93bc-b73c2f09725f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.354596 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.354438 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-data" pod "6a908b4b-a058-4eea-9d4b-05899552a98c" (UID: "6a908b4b-a058-4eea-9d4b-05899552a98c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.354596 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.354475 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-socket" pod "6a908b4b-a058-4eea-9d4b-05899552a98c" (UID: "6a908b4b-a058-4eea-9d4b-05899552a98c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.354596 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.354568 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a24180-c194-4ead-810e-5e20c52cfbb9-istiod-ca-cert" pod "c2a24180-c194-4ead-810e-5e20c52cfbb9" (UID: "c2a24180-c194-4ead-810e-5e20c52cfbb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:38.354772 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.354621 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a908b4b-a058-4eea-9d4b-05899552a98c-istiod-ca-cert" pod "6a908b4b-a058-4eea-9d4b-05899552a98c" (UID: "6a908b4b-a058-4eea-9d4b-05899552a98c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:38.355284 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.355263 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-data" pod "c2a24180-c194-4ead-810e-5e20c52cfbb9" (UID: "c2a24180-c194-4ead-810e-5e20c52cfbb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.365681 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.365634 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-envoy" pod "43948c14-233c-4679-93bc-b73c2f09725f" (UID: "43948c14-233c-4679-93bc-b73c2f09725f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:38.382908 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.382869 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43948c14-233c-4679-93bc-b73c2f09725f-istio-podinfo" pod "43948c14-233c-4679-93bc-b73c2f09725f" (UID: "43948c14-233c-4679-93bc-b73c2f09725f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:38.383635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.382938 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-kube-api-access-66zbt" pod "6a908b4b-a058-4eea-9d4b-05899552a98c" (UID: "6a908b4b-a058-4eea-9d4b-05899552a98c"). InnerVolumeSpecName "kube-api-access-66zbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:38.383635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.382923 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-token" pod "c2a24180-c194-4ead-810e-5e20c52cfbb9" (UID: "c2a24180-c194-4ead-810e-5e20c52cfbb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:38.383830 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.383645 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-kube-api-access-t6pw5" pod "43948c14-233c-4679-93bc-b73c2f09725f" (UID: "43948c14-233c-4679-93bc-b73c2f09725f"). InnerVolumeSpecName "kube-api-access-t6pw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:38.383915 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.383886 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-podinfo" pod "c2a24180-c194-4ead-810e-5e20c52cfbb9" (UID: "c2a24180-c194-4ead-810e-5e20c52cfbb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:38.384008 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.383980 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-kube-api-access-945ph" pod "c2a24180-c194-4ead-810e-5e20c52cfbb9" (UID: "c2a24180-c194-4ead-810e-5e20c52cfbb9"). InnerVolumeSpecName "kube-api-access-945ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:38.384290 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.384087 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-token" pod "6a908b4b-a058-4eea-9d4b-05899552a98c" (UID: "6a908b4b-a058-4eea-9d4b-05899552a98c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:38.451823 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451787 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.451823 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451823 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451836 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451850 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451864 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451878 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a908b4b-a058-4eea-9d4b-05899552a98c-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451892 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451904 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451916 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6pw5\" (UniqueName: \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-kube-api-access-t6pw5\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451931 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-945ph\" (UniqueName: \"kubernetes.io/projected/c2a24180-c194-4ead-810e-5e20c52cfbb9-kube-api-access-945ph\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451945 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451960 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a24180-c194-4ead-810e-5e20c52cfbb9-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451976 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66zbt\" (UniqueName: \"kubernetes.io/projected/6a908b4b-a058-4eea-9d4b-05899552a98c-kube-api-access-66zbt\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.451990 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452006 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452019 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452032 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452043 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43948c14-233c-4679-93bc-b73c2f09725f-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452056 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a908b4b-a058-4eea-9d4b-05899552a98c-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452068 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452080 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452088 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452096 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43948c14-233c-4679-93bc-b73c2f09725f-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452104 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a24180-c194-4ead-810e-5e20c52cfbb9-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:38.452361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:38.452112 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43948c14-233c-4679-93bc-b73c2f09725f-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:39.142989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.142887 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" event={"ID":"6a908b4b-a058-4eea-9d4b-05899552a98c","Type":"ContainerDied","Data":"680aefbf56c73e1e7b900d58adfcfe9974b6f26b6afced0143d3ece43f24f2ee"} Nov 03 06:11:39.142989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.142906 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll" Nov 03 06:11:39.142989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.142955 4855 scope.go:122] "RemoveContainer" containerID="16dd80b96b3f0e0b29eedb81d1b6667865540f07bc45c8a0d92016583bd07747" Nov 03 06:11:39.144499 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.144479 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" Nov 03 06:11:39.144631 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.144506 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n" event={"ID":"43948c14-233c-4679-93bc-b73c2f09725f","Type":"ContainerDied","Data":"57db912b201882eaa3cc09a3184dfc800bd780843aa87056b9b92c78675ef0cd"} Nov 03 06:11:39.146094 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.146069 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" event={"ID":"87cee8e8-09ff-49e8-89ee-04c82a0f50a9","Type":"ContainerDied","Data":"4952e16084fb6f4adf86caef08b99ec385310180f3323d9d53028423a22f8d4a"} Nov 03 06:11:39.146094 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.146089 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb" Nov 03 06:11:39.147677 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.147655 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" Nov 03 06:11:39.147777 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.147678 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk" event={"ID":"c2a24180-c194-4ead-810e-5e20c52cfbb9","Type":"ContainerDied","Data":"db5d1b4cbe1b774bc7a52f0483d0fbfd7aee4bd09cad55a464bc5e22f8ebd8e9"} Nov 03 06:11:39.156907 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.156512 4855 scope.go:122] "RemoveContainer" containerID="7d20f21835eee29c53b5ffda686806674f84e6264eb08f88bbb34c6a7fdd7a62" Nov 03 06:11:39.166090 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.166069 4855 scope.go:122] "RemoveContainer" containerID="7794b81a70c72b7b909cf1adeb6035a3765690ea4c1c4c9287e1d02b472955cc" Nov 03 06:11:39.173977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.173961 4855 scope.go:122] "RemoveContainer" containerID="7e0e80823f4ba203a79b6527166f92c61ec0449913fb8cea2de4587069cc6abc" Nov 03 06:11:39.216944 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.216920 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll"] Nov 03 06:11:39.230169 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.230139 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-f7xll"] Nov 03 06:11:39.275695 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.275668 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk"] Nov 03 06:11:39.281760 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.281736 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-zc2pk"] Nov 03 06:11:39.323869 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.323839 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb"] Nov 03 06:11:39.331322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.331299 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-krccb"] Nov 03 06:11:39.376280 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.376254 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n"] Nov 03 06:11:39.379914 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:39.379892 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-njn9n"] Nov 03 06:11:40.332607 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:40.332572 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43948c14-233c-4679-93bc-b73c2f09725f" path="/var/lib/kubelet/pods/43948c14-233c-4679-93bc-b73c2f09725f/volumes" Nov 03 06:11:40.333209 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:40.333193 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" path="/var/lib/kubelet/pods/6a908b4b-a058-4eea-9d4b-05899552a98c/volumes" Nov 03 06:11:40.333673 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:40.333659 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" path="/var/lib/kubelet/pods/87cee8e8-09ff-49e8-89ee-04c82a0f50a9/volumes" Nov 03 06:11:40.334100 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:40.334086 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" path="/var/lib/kubelet/pods/c2a24180-c194-4ead-810e-5e20c52cfbb9/volumes" Nov 03 06:11:42.403510 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:42.403472 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd"] Nov 03 06:11:42.403931 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:42.403672 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" containerID="cri-o://b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a" gracePeriod=30 Nov 03 06:11:42.417029 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:42.417001 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml"] Nov 03 06:11:42.417231 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:42.417209 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" containerID="cri-o://6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e" gracePeriod=30 Nov 03 06:11:42.443761 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:42.443734 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh"] Nov 03 06:11:42.443941 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:42.443923 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" containerID="cri-o://24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb" gracePeriod=30 Nov 03 06:11:42.445948 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:42.445918 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz"] Nov 03 06:11:42.446091 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:42.446071 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" containerID="cri-o://dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310" gracePeriod=30 Nov 03 06:11:42.737937 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:42.737847 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:42.737937 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:42.737911 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:44.680266 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:44.680234 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:11:46.439788 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:46.439755 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:46.440262 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:46.439810 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:47.240989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.240951 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-678d788c65-2vlhl"] Nov 03 06:11:47.241242 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.241215 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-678d788c65-2vlhl" podUID="878cafe2-86a7-4331-9a14-9874117a22db" containerName="manager" containerID="cri-o://743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d" gracePeriod=30 Nov 03 06:11:47.391039 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.390973 4855 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:11:47.391217 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.391051 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:11:47.671896 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.671783 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:11:47.706184 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.706145 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:11:47.755654 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.755581 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:11:47.766818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.766782 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-envoy\") pod \"5507309e-a297-480d-bd20-4d3914fe9cae\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " Nov 03 06:11:47.767001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.766882 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-istio-token\" (UniqueName: \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-istio-token\") pod \"5507309e-a297-480d-bd20-4d3914fe9cae\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " Nov 03 06:11:47.767001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.766924 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-data\") pod \"5507309e-a297-480d-bd20-4d3914fe9cae\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " Nov 03 06:11:47.767001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.766965 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-kube-api-access-67kbq\" (UniqueName: \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-kube-api-access-67kbq\") pod \"5507309e-a297-480d-bd20-4d3914fe9cae\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " Nov 03 06:11:47.767176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.767050 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-socket\") pod \"5507309e-a297-480d-bd20-4d3914fe9cae\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " Nov 03 06:11:47.767176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.767088 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-webhook-cert\" (UniqueName: \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-webhook-cert\") pod \"878cafe2-86a7-4331-9a14-9874117a22db\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " Nov 03 06:11:47.767176 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.767149 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5507309e-a297-480d-bd20-4d3914fe9cae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5507309e-a297-480d-bd20-4d3914fe9cae-istiod-ca-cert\") pod \"5507309e-a297-480d-bd20-4d3914fe9cae\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " Nov 03 06:11:47.767341 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.767188 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-metrics-certs\") pod \"878cafe2-86a7-4331-9a14-9874117a22db\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " Nov 03 06:11:47.767341 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.767268 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-data" pod "5507309e-a297-480d-bd20-4d3914fe9cae" (UID: "5507309e-a297-480d-bd20-4d3914fe9cae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.767341 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.767297 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/878cafe2-86a7-4331-9a14-9874117a22db-kube-api-access-xdq8d\" (UniqueName: \"kubernetes.io/projected/878cafe2-86a7-4331-9a14-9874117a22db-kube-api-access-xdq8d\") pod \"878cafe2-86a7-4331-9a14-9874117a22db\" (UID: \"878cafe2-86a7-4331-9a14-9874117a22db\") " Nov 03 06:11:47.767341 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.767338 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-certs\") pod \"5507309e-a297-480d-bd20-4d3914fe9cae\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " Nov 03 06:11:47.767635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.767371 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5507309e-a297-480d-bd20-4d3914fe9cae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5507309e-a297-480d-bd20-4d3914fe9cae-istio-podinfo\") pod \"5507309e-a297-480d-bd20-4d3914fe9cae\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " Nov 03 06:11:47.767635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.767478 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-credential-socket\") pod \"5507309e-a297-480d-bd20-4d3914fe9cae\" (UID: \"5507309e-a297-480d-bd20-4d3914fe9cae\") " Nov 03 06:11:47.767940 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.767914 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.768228 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.768201 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-credential-socket" pod "5507309e-a297-480d-bd20-4d3914fe9cae" (UID: "5507309e-a297-480d-bd20-4d3914fe9cae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.768459 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.768418 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-socket" pod "5507309e-a297-480d-bd20-4d3914fe9cae" (UID: "5507309e-a297-480d-bd20-4d3914fe9cae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.768630 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.768596 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-certs" pod "5507309e-a297-480d-bd20-4d3914fe9cae" (UID: "5507309e-a297-480d-bd20-4d3914fe9cae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.768787 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.768768 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5507309e-a297-480d-bd20-4d3914fe9cae-istiod-ca-cert" pod "5507309e-a297-480d-bd20-4d3914fe9cae" (UID: "5507309e-a297-480d-bd20-4d3914fe9cae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:47.775254 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.775226 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-metrics-certs" pod "878cafe2-86a7-4331-9a14-9874117a22db" (UID: "878cafe2-86a7-4331-9a14-9874117a22db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:11:47.775365 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.775311 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-envoy" pod "5507309e-a297-480d-bd20-4d3914fe9cae" (UID: "5507309e-a297-480d-bd20-4d3914fe9cae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.775461 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.775404 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/878cafe2-86a7-4331-9a14-9874117a22db-kube-api-access-xdq8d" pod "878cafe2-86a7-4331-9a14-9874117a22db" (UID: "878cafe2-86a7-4331-9a14-9874117a22db"). InnerVolumeSpecName "kube-api-access-xdq8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:47.775461 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.775379 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-webhook-cert" pod "878cafe2-86a7-4331-9a14-9874117a22db" (UID: "878cafe2-86a7-4331-9a14-9874117a22db"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:11:47.775588 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.775553 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-kube-api-access-67kbq" pod "5507309e-a297-480d-bd20-4d3914fe9cae" (UID: "5507309e-a297-480d-bd20-4d3914fe9cae"). InnerVolumeSpecName "kube-api-access-67kbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:47.777116 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.777086 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5507309e-a297-480d-bd20-4d3914fe9cae-istio-podinfo" pod "5507309e-a297-480d-bd20-4d3914fe9cae" (UID: "5507309e-a297-480d-bd20-4d3914fe9cae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:47.778003 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.777967 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-istio-token" pod "5507309e-a297-480d-bd20-4d3914fe9cae" (UID: "5507309e-a297-480d-bd20-4d3914fe9cae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:47.798559 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.798524 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:11:47.800907 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.800882 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:11:47.868957 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.868923 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea987416-f4be-475c-8574-903796fcf95d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea987416-f4be-475c-8574-903796fcf95d-istiod-ca-cert\") pod \"ea987416-f4be-475c-8574-903796fcf95d\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " Nov 03 06:11:47.869163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.868977 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-certs\") pod \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " Nov 03 06:11:47.869163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869037 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-token\" (UniqueName: \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-token\") pod \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " Nov 03 06:11:47.869163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869089 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-credential-socket\") pod \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " Nov 03 06:11:47.869163 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869126 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-kube-api-access-tpwc6\" (UniqueName: \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-kube-api-access-tpwc6\") pod \"9fdb858d-e18c-4d4b-abce-054fe2219165\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " Nov 03 06:11:47.869408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869174 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-data\") pod \"9fdb858d-e18c-4d4b-abce-054fe2219165\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " Nov 03 06:11:47.869408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869203 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-socket\") pod \"9fdb858d-e18c-4d4b-abce-054fe2219165\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " Nov 03 06:11:47.869408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869237 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-credential-socket\") pod \"9fdb858d-e18c-4d4b-abce-054fe2219165\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " Nov 03 06:11:47.869408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869275 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-socket\") pod \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " Nov 03 06:11:47.869408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869308 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-envoy\") pod \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " Nov 03 06:11:47.869408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869334 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-envoy\") pod \"9fdb858d-e18c-4d4b-abce-054fe2219165\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " Nov 03 06:11:47.869408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869334 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-certs" pod "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" (UID: "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.869408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869373 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-envoy\") pod \"ea987416-f4be-475c-8574-903796fcf95d\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " Nov 03 06:11:47.869815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869443 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-kube-api-access-g9sft\" (UniqueName: \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-kube-api-access-g9sft\") pod \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " Nov 03 06:11:47.869815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869474 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-podinfo\") pod \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " Nov 03 06:11:47.869815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869505 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-credential-socket\") pod \"ea987416-f4be-475c-8574-903796fcf95d\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " Nov 03 06:11:47.869815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869534 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-token\" (UniqueName: \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-token\") pod \"9fdb858d-e18c-4d4b-abce-054fe2219165\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " Nov 03 06:11:47.869815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869573 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-podinfo\") pod \"9fdb858d-e18c-4d4b-abce-054fe2219165\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " Nov 03 06:11:47.869815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869620 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-certs\") pod \"ea987416-f4be-475c-8574-903796fcf95d\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " Nov 03 06:11:47.869815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869649 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-socket\") pod \"ea987416-f4be-475c-8574-903796fcf95d\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " Nov 03 06:11:47.869815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869682 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-data\") pod \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " Nov 03 06:11:47.869815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869731 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istiod-ca-cert\") pod \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\" (UID: \"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349\") " Nov 03 06:11:47.869815 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869778 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-kube-api-access-zbbcb\" (UniqueName: \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-kube-api-access-zbbcb\") pod \"ea987416-f4be-475c-8574-903796fcf95d\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " Nov 03 06:11:47.870318 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869823 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-data\") pod \"ea987416-f4be-475c-8574-903796fcf95d\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " Nov 03 06:11:47.870318 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869851 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea987416-f4be-475c-8574-903796fcf95d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea987416-f4be-475c-8574-903796fcf95d-istio-podinfo\") pod \"ea987416-f4be-475c-8574-903796fcf95d\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " Nov 03 06:11:47.870318 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869880 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-certs\") pod \"9fdb858d-e18c-4d4b-abce-054fe2219165\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " Nov 03 06:11:47.870318 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869911 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-istio-token\" (UniqueName: \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-istio-token\") pod \"ea987416-f4be-475c-8574-903796fcf95d\" (UID: \"ea987416-f4be-475c-8574-903796fcf95d\") " Nov 03 06:11:47.870318 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869948 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fdb858d-e18c-4d4b-abce-054fe2219165-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fdb858d-e18c-4d4b-abce-054fe2219165-istiod-ca-cert\") pod \"9fdb858d-e18c-4d4b-abce-054fe2219165\" (UID: \"9fdb858d-e18c-4d4b-abce-054fe2219165\") " Nov 03 06:11:47.870630 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870603 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870687 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870631 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870687 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870646 4855 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-webhook-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870687 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870661 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5507309e-a297-480d-bd20-4d3914fe9cae-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870687 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870674 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878cafe2-86a7-4331-9a14-9874117a22db-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870688 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdq8d\" (UniqueName: \"kubernetes.io/projected/878cafe2-86a7-4331-9a14-9874117a22db-kube-api-access-xdq8d\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870704 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870719 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5507309e-a297-480d-bd20-4d3914fe9cae-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870733 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870747 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5507309e-a297-480d-bd20-4d3914fe9cae-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870761 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.870882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870811 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67kbq\" (UniqueName: \"kubernetes.io/projected/5507309e-a297-480d-bd20-4d3914fe9cae-kube-api-access-67kbq\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869370 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea987416-f4be-475c-8574-903796fcf95d-istiod-ca-cert" pod "ea987416-f4be-475c-8574-903796fcf95d" (UID: "ea987416-f4be-475c-8574-903796fcf95d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869504 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-credential-socket" pod "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" (UID: "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869574 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-socket" pod "9fdb858d-e18c-4d4b-abce-054fe2219165" (UID: "9fdb858d-e18c-4d4b-abce-054fe2219165"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869664 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-socket" pod "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" (UID: "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.869836 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-credential-socket" pod "9fdb858d-e18c-4d4b-abce-054fe2219165" (UID: "9fdb858d-e18c-4d4b-abce-054fe2219165"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870032 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-certs" pod "ea987416-f4be-475c-8574-903796fcf95d" (UID: "ea987416-f4be-475c-8574-903796fcf95d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870066 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-data" pod "9fdb858d-e18c-4d4b-abce-054fe2219165" (UID: "9fdb858d-e18c-4d4b-abce-054fe2219165"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.870889 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-credential-socket" pod "ea987416-f4be-475c-8574-903796fcf95d" (UID: "ea987416-f4be-475c-8574-903796fcf95d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.871111 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-data" pod "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" (UID: "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.871284 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-socket" pod "ea987416-f4be-475c-8574-903796fcf95d" (UID: "ea987416-f4be-475c-8574-903796fcf95d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.871442 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istiod-ca-cert" pod "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" (UID: "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.871600 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdb858d-e18c-4d4b-abce-054fe2219165-istiod-ca-cert" pod "9fdb858d-e18c-4d4b-abce-054fe2219165" (UID: "9fdb858d-e18c-4d4b-abce-054fe2219165"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.871893 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-certs" pod "9fdb858d-e18c-4d4b-abce-054fe2219165" (UID: "9fdb858d-e18c-4d4b-abce-054fe2219165"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.871968 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-data" pod "ea987416-f4be-475c-8574-903796fcf95d" (UID: "ea987416-f4be-475c-8574-903796fcf95d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.872939 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.872530 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-token" pod "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" (UID: "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:47.875132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.875048 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-envoy" pod "ea987416-f4be-475c-8574-903796fcf95d" (UID: "ea987416-f4be-475c-8574-903796fcf95d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.875132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.875065 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-kube-api-access-zbbcb" pod "ea987416-f4be-475c-8574-903796fcf95d" (UID: "ea987416-f4be-475c-8574-903796fcf95d"). InnerVolumeSpecName "kube-api-access-zbbcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:47.875132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.875079 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-kube-api-access-g9sft" pod "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" (UID: "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349"). InnerVolumeSpecName "kube-api-access-g9sft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:47.875132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.875087 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-token" pod "9fdb858d-e18c-4d4b-abce-054fe2219165" (UID: "9fdb858d-e18c-4d4b-abce-054fe2219165"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:47.875424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.875121 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea987416-f4be-475c-8574-903796fcf95d-istio-podinfo" pod "ea987416-f4be-475c-8574-903796fcf95d" (UID: "ea987416-f4be-475c-8574-903796fcf95d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:47.875424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.875182 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-istio-token" pod "ea987416-f4be-475c-8574-903796fcf95d" (UID: "ea987416-f4be-475c-8574-903796fcf95d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:47.875424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.875264 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-podinfo" pod "9fdb858d-e18c-4d4b-abce-054fe2219165" (UID: "9fdb858d-e18c-4d4b-abce-054fe2219165"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:47.875712 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.875694 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-kube-api-access-tpwc6" pod "9fdb858d-e18c-4d4b-abce-054fe2219165" (UID: "9fdb858d-e18c-4d4b-abce-054fe2219165"). InnerVolumeSpecName "kube-api-access-tpwc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:11:47.875712 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.875708 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-podinfo" pod "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" (UID: "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:11:47.876152 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.876135 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-envoy" pod "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" (UID: "8ad38dc0-f80c-4c16-bd8f-ecb23a35c349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.877080 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.877063 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-envoy" pod "9fdb858d-e18c-4d4b-abce-054fe2219165" (UID: "9fdb858d-e18c-4d4b-abce-054fe2219165"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:11:47.972253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972213 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9sft\" (UniqueName: \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-kube-api-access-g9sft\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972244 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972257 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972271 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972284 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972294 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972304 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972316 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972328 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972341 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbbcb\" (UniqueName: \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-kube-api-access-zbbcb\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972353 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972365 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea987416-f4be-475c-8574-903796fcf95d-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972375 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972401 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea987416-f4be-475c-8574-903796fcf95d-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972413 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fdb858d-e18c-4d4b-abce-054fe2219165-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972424 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea987416-f4be-475c-8574-903796fcf95d-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972436 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972447 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972459 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpwc6\" (UniqueName: \"kubernetes.io/projected/9fdb858d-e18c-4d4b-abce-054fe2219165-kube-api-access-tpwc6\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972471 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972484 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972495 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972507 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972520 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972532 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fdb858d-e18c-4d4b-abce-054fe2219165-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:47.972605 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:47.972545 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea987416-f4be-475c-8574-903796fcf95d-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:11:48.198171 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.198138 4855 generic.go:356] "Generic (PLEG): container finished" podID="878cafe2-86a7-4331-9a14-9874117a22db" containerID="743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d" exitCode=0 Nov 03 06:11:48.198336 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.198208 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-678d788c65-2vlhl" Nov 03 06:11:48.198336 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.198211 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-678d788c65-2vlhl" event={"ID":"878cafe2-86a7-4331-9a14-9874117a22db","Type":"ContainerDied","Data":"743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d"} Nov 03 06:11:48.198336 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.198296 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-678d788c65-2vlhl" event={"ID":"878cafe2-86a7-4331-9a14-9874117a22db","Type":"ContainerDied","Data":"2bcfca4b489c79dd151e8c80b66638298b9bda36d183721bfb2283bb399ac91e"} Nov 03 06:11:48.198336 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.198324 4855 scope.go:122] "RemoveContainer" containerID="743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d" Nov 03 06:11:48.199610 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.199586 4855 generic.go:356] "Generic (PLEG): container finished" podID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerID="dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310" exitCode=0 Nov 03 06:11:48.199744 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.199663 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" Nov 03 06:11:48.199744 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.199676 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" event={"ID":"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349","Type":"ContainerDied","Data":"dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310"} Nov 03 06:11:48.199744 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.199713 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz" event={"ID":"8ad38dc0-f80c-4c16-bd8f-ecb23a35c349","Type":"ContainerDied","Data":"96359097855bc2987eeb77f5cf2bdcb3e51e1af4ff63c856eb89e985cabde811"} Nov 03 06:11:48.201112 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.201085 4855 generic.go:356] "Generic (PLEG): container finished" podID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerID="b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a" exitCode=0 Nov 03 06:11:48.201246 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.201117 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" event={"ID":"9fdb858d-e18c-4d4b-abce-054fe2219165","Type":"ContainerDied","Data":"b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a"} Nov 03 06:11:48.201246 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.201140 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" Nov 03 06:11:48.201246 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.201159 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd" event={"ID":"9fdb858d-e18c-4d4b-abce-054fe2219165","Type":"ContainerDied","Data":"a3371c995dc65dbe0a7eb7cbb70c6a0b12702392d95bcd55fbca7bb644865478"} Nov 03 06:11:48.203101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.203075 4855 generic.go:356] "Generic (PLEG): container finished" podID="5507309e-a297-480d-bd20-4d3914fe9cae" containerID="6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e" exitCode=0 Nov 03 06:11:48.203207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.203101 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" event={"ID":"5507309e-a297-480d-bd20-4d3914fe9cae","Type":"ContainerDied","Data":"6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e"} Nov 03 06:11:48.203207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.203146 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" Nov 03 06:11:48.203320 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.203151 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml" event={"ID":"5507309e-a297-480d-bd20-4d3914fe9cae","Type":"ContainerDied","Data":"5084a3f4c4486861323e9dfa81ba549a751f7e75d5a691ca39955ff85da22a7f"} Nov 03 06:11:48.204754 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.204734 4855 generic.go:356] "Generic (PLEG): container finished" podID="ea987416-f4be-475c-8574-903796fcf95d" containerID="24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb" exitCode=0 Nov 03 06:11:48.204851 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.204786 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" Nov 03 06:11:48.204851 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.204809 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" event={"ID":"ea987416-f4be-475c-8574-903796fcf95d","Type":"ContainerDied","Data":"24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb"} Nov 03 06:11:48.204851 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.204840 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh" event={"ID":"ea987416-f4be-475c-8574-903796fcf95d","Type":"ContainerDied","Data":"5e79946052c04d120ee141443c3fdaccf12a12440b35987d0edca9ceaebcbbfc"} Nov 03 06:11:48.210168 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.210011 4855 scope.go:122] "RemoveContainer" containerID="743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d" Nov 03 06:11:48.210531 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:48.210377 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d\": container with ID starting with 743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d not found: ID does not exist" containerID="743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d" Nov 03 06:11:48.210531 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.210438 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d"} err="failed to get container status \"743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d\": rpc error: code = NotFound desc = could not find container \"743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d\": container with ID starting with 743310c63053d14911b59d8ceee6297d2c42b5f8553b973ce11f7b1b9b0f9d4d not found: ID does not exist" Nov 03 06:11:48.210531 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.210461 4855 scope.go:122] "RemoveContainer" containerID="dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310" Nov 03 06:11:48.222001 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.221983 4855 scope.go:122] "RemoveContainer" containerID="dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310" Nov 03 06:11:48.222251 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:48.222232 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310\": container with ID starting with dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310 not found: ID does not exist" containerID="dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310" Nov 03 06:11:48.222316 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.222258 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310"} err="failed to get container status \"dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310\": rpc error: code = NotFound desc = could not find container \"dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310\": container with ID starting with dec7c808e12894f658e8ba52eb2bedd1951424aa493c2f1357159f684a143310 not found: ID does not exist" Nov 03 06:11:48.222316 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.222273 4855 scope.go:122] "RemoveContainer" containerID="b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a" Nov 03 06:11:48.230626 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.230602 4855 scope.go:122] "RemoveContainer" containerID="b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a" Nov 03 06:11:48.230875 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:48.230856 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a\": container with ID starting with b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a not found: ID does not exist" containerID="b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a" Nov 03 06:11:48.230933 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.230881 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a"} err="failed to get container status \"b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a\": rpc error: code = NotFound desc = could not find container \"b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a\": container with ID starting with b0417c855631135109a999546af2dd6f52bba6a14308662f856f24edc989cc9a not found: ID does not exist" Nov 03 06:11:48.230933 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.230900 4855 scope.go:122] "RemoveContainer" containerID="6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e" Nov 03 06:11:48.239077 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.239059 4855 scope.go:122] "RemoveContainer" containerID="5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53" Nov 03 06:11:48.247679 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.247532 4855 scope.go:122] "RemoveContainer" containerID="6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e" Nov 03 06:11:48.247821 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:48.247789 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e\": container with ID starting with 6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e not found: ID does not exist" containerID="6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e" Nov 03 06:11:48.247878 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.247829 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e"} err="failed to get container status \"6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e\": rpc error: code = NotFound desc = could not find container \"6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e\": container with ID starting with 6d8675e3b72eb920adb12384a4a94c91507913305e248c92393ad60aa1fe743e not found: ID does not exist" Nov 03 06:11:48.247878 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.247845 4855 scope.go:122] "RemoveContainer" containerID="5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53" Nov 03 06:11:48.248119 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:48.248097 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53\": container with ID starting with 5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53 not found: ID does not exist" containerID="5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53" Nov 03 06:11:48.248179 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.248127 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53"} err="failed to get container status \"5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53\": rpc error: code = NotFound desc = could not find container \"5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53\": container with ID starting with 5d9abbbc2361d534dd0ca339051fce5b7853ca1e2115321ffea182b6779d4f53 not found: ID does not exist" Nov 03 06:11:48.248179 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.248146 4855 scope.go:122] "RemoveContainer" containerID="24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb" Nov 03 06:11:48.257200 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.257178 4855 scope.go:122] "RemoveContainer" containerID="24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb" Nov 03 06:11:48.257567 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:48.257546 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb\": container with ID starting with 24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb not found: ID does not exist" containerID="24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb" Nov 03 06:11:48.257639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.257572 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb"} err="failed to get container status \"24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb\": rpc error: code = NotFound desc = could not find container \"24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb\": container with ID starting with 24e8477794d82f0cb9ee0760c57d80a0ed2ade94e5017d0a304d46b5acad52fb not found: ID does not exist" Nov 03 06:11:48.258079 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.258062 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-678d788c65-2vlhl"] Nov 03 06:11:48.271665 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.271643 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-678d788c65-2vlhl"] Nov 03 06:11:48.331764 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.331731 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="878cafe2-86a7-4331-9a14-9874117a22db" path="/var/lib/kubelet/pods/878cafe2-86a7-4331-9a14-9874117a22db/volumes" Nov 03 06:11:48.376580 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.376550 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh"] Nov 03 06:11:48.379190 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.379165 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-vv5sh"] Nov 03 06:11:48.417230 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.417203 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml"] Nov 03 06:11:48.422831 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.422805 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-sm7ml"] Nov 03 06:11:48.461070 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.460985 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz"] Nov 03 06:11:48.462528 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.462506 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-lr5rz"] Nov 03 06:11:48.500491 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.500462 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd"] Nov 03 06:11:48.506198 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.506173 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-hsvmd"] Nov 03 06:11:48.968146 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968111 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-678d788c65-5rsst"] Nov 03 06:11:48.968688 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968667 4855 cpu_manager.go:414] "containerMap: removing container" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" Nov 03 06:11:48.968750 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968689 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" Nov 03 06:11:48.968750 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968704 4855 cpu_manager.go:414] "containerMap: removing container" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" Nov 03 06:11:48.968750 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968712 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" Nov 03 06:11:48.968750 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968731 4855 cpu_manager.go:414] "containerMap: removing container" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" Nov 03 06:11:48.968750 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968739 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968754 4855 cpu_manager.go:414] "containerMap: removing container" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968761 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968770 4855 cpu_manager.go:414] "containerMap: removing container" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968777 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968791 4855 cpu_manager.go:414] "containerMap: removing container" podUID="97ba4903-d54a-4291-bd8b-811bc6b3302a" containerName="payload-pre-processing" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968798 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ba4903-d54a-4291-bd8b-811bc6b3302a" containerName="payload-pre-processing" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968810 4855 cpu_manager.go:414] "containerMap: removing container" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968817 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968826 4855 cpu_manager.go:414] "containerMap: removing container" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968833 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968842 4855 cpu_manager.go:414] "containerMap: removing container" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968849 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968866 4855 cpu_manager.go:414] "containerMap: removing container" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968874 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968883 4855 cpu_manager.go:414] "containerMap: removing container" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968890 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968903 4855 cpu_manager.go:414] "containerMap: removing container" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968909 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968922 4855 cpu_manager.go:414] "containerMap: removing container" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968929 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968941 4855 cpu_manager.go:414] "containerMap: removing container" podUID="878cafe2-86a7-4331-9a14-9874117a22db" containerName="manager" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968947 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="878cafe2-86a7-4331-9a14-9874117a22db" containerName="manager" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968957 4855 cpu_manager.go:414] "containerMap: removing container" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968964 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968976 4855 cpu_manager.go:414] "containerMap: removing container" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968983 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.968994 4855 cpu_manager.go:414] "containerMap: removing container" podUID="5809e76e-9af0-4889-bf3c-5b9b7102eb1d" containerName="maas-api" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969001 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5809e76e-9af0-4889-bf3c-5b9b7102eb1d" containerName="maas-api" Nov 03 06:11:48.968999 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969012 4855 cpu_manager.go:414] "containerMap: removing container" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969019 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969032 4855 cpu_manager.go:414] "containerMap: removing container" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969039 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969053 4855 cpu_manager.go:414] "containerMap: removing container" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969059 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969069 4855 cpu_manager.go:414] "containerMap: removing container" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969075 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969085 4855 cpu_manager.go:414] "containerMap: removing container" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969092 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969109 4855 cpu_manager.go:414] "containerMap: removing container" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969116 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969128 4855 cpu_manager.go:414] "containerMap: removing container" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969134 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969143 4855 cpu_manager.go:414] "containerMap: removing container" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969151 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969166 4855 cpu_manager.go:414] "containerMap: removing container" podUID="34d1643a-db61-492d-8b12-ef21e5eb0834" containerName="maas-api" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969172 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d1643a-db61-492d-8b12-ef21e5eb0834" containerName="maas-api" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969183 4855 cpu_manager.go:414] "containerMap: removing container" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969190 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969202 4855 cpu_manager.go:414] "containerMap: removing container" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969209 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969221 4855 cpu_manager.go:414] "containerMap: removing container" podUID="ea38a440-6197-4e41-b316-955e76211942" containerName="maas-api" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969227 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea38a440-6197-4e41-b316-955e76211942" containerName="maas-api" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969240 4855 cpu_manager.go:414] "containerMap: removing container" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969246 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969257 4855 cpu_manager.go:414] "containerMap: removing container" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969264 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969273 4855 cpu_manager.go:414] "containerMap: removing container" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969280 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969291 4855 cpu_manager.go:414] "containerMap: removing container" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969298 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969405 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969417 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="21990fba-a699-4ba0-8a0d-ac9025d3c129" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969428 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2e8f6c-d325-4e8c-9f49-93477681daab" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969441 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ba4903-d54a-4291-bd8b-811bc6b3302a" containerName="payload-pre-processing" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969451 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969458 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e65c31d-5628-4811-a84e-405aa9eb9188" containerName="main" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969468 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3ebfb1-a8b8-46c7-abcd-f24e6b383a62" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969475 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a908b4b-a058-4eea-9d4b-05899552a98c" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969483 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b19d23e-3efa-4c72-a703-6610bd96a3c4" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969491 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6892c41-6866-43ee-82da-594bc4e28b29" containerName="main" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969499 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5c4df2-55ce-4bd3-b10d-10ba796dce4c" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969508 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d1643a-db61-492d-8b12-ef21e5eb0834" containerName="maas-api" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969518 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafef133-515b-4fdf-a799-cf10a42224f8" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969526 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f0b7ce-07b2-43be-819f-cde40d902902" containerName="main" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969535 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea38a440-6197-4e41-b316-955e76211942" containerName="maas-api" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969541 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="43948c14-233c-4679-93bc-b73c2f09725f" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969548 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48278d0-aa0f-4f79-af81-39f95b6c5495" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969557 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fbec74-f9e1-4d46-abcc-cedad8e517fd" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969565 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea987416-f4be-475c-8574-903796fcf95d" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969574 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee92591f-8473-4981-8af8-bb60f096c653" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969582 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969592 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="878cafe2-86a7-4331-9a14-9874117a22db" containerName="manager" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969600 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d926084c-dafb-4fd7-adc3-928dc9036a92" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969607 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80d1e19-3ce6-4060-8305-7cfec3f4c296" containerName="main" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969615 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969622 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="82859956-f5dd-43be-894b-2e78907f7505" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969631 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="5809e76e-9af0-4889-bf3c-5b9b7102eb1d" containerName="maas-api" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969638 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969647 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="299590cc-de3d-4408-8653-320887002525" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969654 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a24180-c194-4ead-810e-5e20c52cfbb9" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969662 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7ea471-1f77-49c0-92dc-139f56db1d14" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969671 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3a0b0e-10e5-4605-aa0e-67745585646d" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969678 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" containerName="istio-proxy" Nov 03 06:11:48.970454 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.969687 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="87cee8e8-09ff-49e8-89ee-04c82a0f50a9" containerName="istio-proxy" Nov 03 06:11:48.976970 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.976948 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:48.997052 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.997017 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4ffnq\"" Nov 03 06:11:48.997187 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.997073 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 06:11:48.997187 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:48.997107 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 06:11:49.012860 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.012834 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-678d788c65-5rsst"] Nov 03 06:11:49.043578 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.043551 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 06:11:49.081982 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.081949 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8wbn\" (UniqueName: \"kubernetes.io/projected/de221a90-6ca4-4606-a09e-1473cab5b01e-kube-api-access-g8wbn\") pod \"maas-controller-678d788c65-5rsst\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:49.082226 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.081992 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-metrics-certs\") pod \"maas-controller-678d788c65-5rsst\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:49.082226 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.082013 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-webhook-cert\") pod \"maas-controller-678d788c65-5rsst\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:49.183447 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.183407 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8wbn\" (UniqueName: \"kubernetes.io/projected/de221a90-6ca4-4606-a09e-1473cab5b01e-kube-api-access-g8wbn\") pod \"maas-controller-678d788c65-5rsst\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:49.183628 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.183469 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-metrics-certs\") pod \"maas-controller-678d788c65-5rsst\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:49.183628 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.183498 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-webhook-cert\") pod \"maas-controller-678d788c65-5rsst\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:49.185980 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.185948 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-webhook-cert\") pod \"maas-controller-678d788c65-5rsst\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:49.186089 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.185990 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-metrics-certs\") pod \"maas-controller-678d788c65-5rsst\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:49.208819 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.208791 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8wbn\" (UniqueName: \"kubernetes.io/projected/de221a90-6ca4-4606-a09e-1473cab5b01e-kube-api-access-g8wbn\") pod \"maas-controller-678d788c65-5rsst\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:49.294631 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.294542 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:49.598844 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:11:49.598815 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde221a90_6ca4_4606_a09e_1473cab5b01e.slice/crio-f05f108ad1600ddc797580ce1fbf012063400035ddebf55a1221b791e3646297 WatchSource:0}: Error finding container f05f108ad1600ddc797580ce1fbf012063400035ddebf55a1221b791e3646297: Status 404 returned error can't find the container with id f05f108ad1600ddc797580ce1fbf012063400035ddebf55a1221b791e3646297 Nov 03 06:11:49.600127 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.600100 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-678d788c65-5rsst"] Nov 03 06:11:49.600205 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:49.600178 4855 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 06:11:50.223192 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:50.223156 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-678d788c65-5rsst" event={"ID":"de221a90-6ca4-4606-a09e-1473cab5b01e","Type":"ContainerStarted","Data":"f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f"} Nov 03 06:11:50.223723 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:50.223204 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-678d788c65-5rsst" event={"ID":"de221a90-6ca4-4606-a09e-1473cab5b01e","Type":"ContainerStarted","Data":"f05f108ad1600ddc797580ce1fbf012063400035ddebf55a1221b791e3646297"} Nov 03 06:11:50.223723 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:50.223279 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:11:50.270093 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:50.269977 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-678d788c65-5rsst" podStartSLOduration=1.8535852940000002 podStartE2EDuration="2.269963525s" totalImagesPullingTime="416.378231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:11:50.2681281 +0000 UTC m=+998.076515369" watchObservedRunningTime="2026-11-03 06:11:50.269963525 +0000 UTC m=+998.078350857" Nov 03 06:11:50.332823 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:50.332788 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" path="/var/lib/kubelet/pods/5507309e-a297-480d-bd20-4d3914fe9cae/volumes" Nov 03 06:11:50.333245 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:50.333232 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad38dc0-f80c-4c16-bd8f-ecb23a35c349" path="/var/lib/kubelet/pods/8ad38dc0-f80c-4c16-bd8f-ecb23a35c349/volumes" Nov 03 06:11:50.333685 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:50.333671 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fdb858d-e18c-4d4b-abce-054fe2219165" path="/var/lib/kubelet/pods/9fdb858d-e18c-4d4b-abce-054fe2219165/volumes" Nov 03 06:11:50.334344 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:11:50.334326 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea987416-f4be-475c-8574-903796fcf95d" path="/var/lib/kubelet/pods/ea987416-f4be-475c-8574-903796fcf95d/volumes" Nov 03 06:11:58.882979 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:11:58.882940 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:12:01.232119 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:01.232089 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:12:19.935535 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:19.935496 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-678d788c65-5rsst"] Nov 03 06:12:19.936035 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:19.935706 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-678d788c65-5rsst" podUID="de221a90-6ca4-4606-a09e-1473cab5b01e" containerName="manager" containerID="cri-o://f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f" gracePeriod=30 Nov 03 06:12:20.185166 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.185144 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:12:20.282578 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.282488 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-metrics-certs\") pod \"de221a90-6ca4-4606-a09e-1473cab5b01e\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " Nov 03 06:12:20.282729 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.282643 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de221a90-6ca4-4606-a09e-1473cab5b01e-kube-api-access-g8wbn\" (UniqueName: \"kubernetes.io/projected/de221a90-6ca4-4606-a09e-1473cab5b01e-kube-api-access-g8wbn\") pod \"de221a90-6ca4-4606-a09e-1473cab5b01e\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " Nov 03 06:12:20.282729 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.282722 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-webhook-cert\") pod \"de221a90-6ca4-4606-a09e-1473cab5b01e\" (UID: \"de221a90-6ca4-4606-a09e-1473cab5b01e\") " Nov 03 06:12:20.284550 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.284509 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-metrics-certs" pod "de221a90-6ca4-4606-a09e-1473cab5b01e" (UID: "de221a90-6ca4-4606-a09e-1473cab5b01e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:12:20.284690 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.284675 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-webhook-cert" pod "de221a90-6ca4-4606-a09e-1473cab5b01e" (UID: "de221a90-6ca4-4606-a09e-1473cab5b01e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:12:20.284760 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.284729 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de221a90-6ca4-4606-a09e-1473cab5b01e-kube-api-access-g8wbn" pod "de221a90-6ca4-4606-a09e-1473cab5b01e" (UID: "de221a90-6ca4-4606-a09e-1473cab5b01e"). InnerVolumeSpecName "kube-api-access-g8wbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:12:20.383685 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.383649 4855 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-webhook-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:12:20.383685 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.383680 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de221a90-6ca4-4606-a09e-1473cab5b01e-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:12:20.383685 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.383690 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8wbn\" (UniqueName: \"kubernetes.io/projected/de221a90-6ca4-4606-a09e-1473cab5b01e-kube-api-access-g8wbn\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:12:20.416528 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.416489 4855 generic.go:356] "Generic (PLEG): container finished" podID="de221a90-6ca4-4606-a09e-1473cab5b01e" containerID="f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f" exitCode=0 Nov 03 06:12:20.416680 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.416550 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-678d788c65-5rsst" Nov 03 06:12:20.416680 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.416548 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-678d788c65-5rsst" event={"ID":"de221a90-6ca4-4606-a09e-1473cab5b01e","Type":"ContainerDied","Data":"f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f"} Nov 03 06:12:20.416680 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.416667 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-678d788c65-5rsst" event={"ID":"de221a90-6ca4-4606-a09e-1473cab5b01e","Type":"ContainerDied","Data":"f05f108ad1600ddc797580ce1fbf012063400035ddebf55a1221b791e3646297"} Nov 03 06:12:20.416787 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.416688 4855 scope.go:122] "RemoveContainer" containerID="f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f" Nov 03 06:12:20.427280 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.427256 4855 scope.go:122] "RemoveContainer" containerID="f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f" Nov 03 06:12:20.427644 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:12:20.427620 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f\": container with ID starting with f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f not found: ID does not exist" containerID="f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f" Nov 03 06:12:20.427747 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.427656 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f"} err="failed to get container status \"f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f\": rpc error: code = NotFound desc = could not find container \"f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f\": container with ID starting with f78f187f27717c6f1e18539c0f85720cbaa0bed58bf594d93313178c1d964d3f not found: ID does not exist" Nov 03 06:12:20.472233 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.472201 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-678d788c65-5rsst"] Nov 03 06:12:20.477081 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:20.477058 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-678d788c65-5rsst"] Nov 03 06:12:21.673110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.673073 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-678d788c65-zrb8q"] Nov 03 06:12:21.673538 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.673446 4855 cpu_manager.go:414] "containerMap: removing container" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" Nov 03 06:12:21.673538 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.673458 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5507309e-a297-480d-bd20-4d3914fe9cae" containerName="istio-proxy" Nov 03 06:12:21.673538 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.673472 4855 cpu_manager.go:414] "containerMap: removing container" podUID="de221a90-6ca4-4606-a09e-1473cab5b01e" containerName="manager" Nov 03 06:12:21.673538 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.673478 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de221a90-6ca4-4606-a09e-1473cab5b01e" containerName="manager" Nov 03 06:12:21.673668 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.673555 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="de221a90-6ca4-4606-a09e-1473cab5b01e" containerName="manager" Nov 03 06:12:21.678103 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.678083 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:21.708035 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.708008 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 06:12:21.708253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.708081 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 06:12:21.708253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.708117 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4ffnq\"" Nov 03 06:12:21.708253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.708170 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 06:12:21.712543 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.712521 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-678d788c65-zrb8q"] Nov 03 06:12:21.796522 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.796489 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66hz\" (UniqueName: \"kubernetes.io/projected/549a963a-6ae5-48c8-a7f2-2935d181e004-kube-api-access-f66hz\") pod \"maas-controller-678d788c65-zrb8q\" (UID: \"549a963a-6ae5-48c8-a7f2-2935d181e004\") " pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:21.796706 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.796555 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/549a963a-6ae5-48c8-a7f2-2935d181e004-webhook-cert\") pod \"maas-controller-678d788c65-zrb8q\" (UID: \"549a963a-6ae5-48c8-a7f2-2935d181e004\") " pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:21.796706 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.796632 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549a963a-6ae5-48c8-a7f2-2935d181e004-metrics-certs\") pod \"maas-controller-678d788c65-zrb8q\" (UID: \"549a963a-6ae5-48c8-a7f2-2935d181e004\") " pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:21.897543 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.897500 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/549a963a-6ae5-48c8-a7f2-2935d181e004-webhook-cert\") pod \"maas-controller-678d788c65-zrb8q\" (UID: \"549a963a-6ae5-48c8-a7f2-2935d181e004\") " pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:21.897738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.897562 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549a963a-6ae5-48c8-a7f2-2935d181e004-metrics-certs\") pod \"maas-controller-678d788c65-zrb8q\" (UID: \"549a963a-6ae5-48c8-a7f2-2935d181e004\") " pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:21.897738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.897666 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66hz\" (UniqueName: \"kubernetes.io/projected/549a963a-6ae5-48c8-a7f2-2935d181e004-kube-api-access-f66hz\") pod \"maas-controller-678d788c65-zrb8q\" (UID: \"549a963a-6ae5-48c8-a7f2-2935d181e004\") " pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:21.900110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.900081 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549a963a-6ae5-48c8-a7f2-2935d181e004-metrics-certs\") pod \"maas-controller-678d788c65-zrb8q\" (UID: \"549a963a-6ae5-48c8-a7f2-2935d181e004\") " pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:21.900231 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.900115 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/549a963a-6ae5-48c8-a7f2-2935d181e004-webhook-cert\") pod \"maas-controller-678d788c65-zrb8q\" (UID: \"549a963a-6ae5-48c8-a7f2-2935d181e004\") " pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:21.922469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.922439 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66hz\" (UniqueName: \"kubernetes.io/projected/549a963a-6ae5-48c8-a7f2-2935d181e004-kube-api-access-f66hz\") pod \"maas-controller-678d788c65-zrb8q\" (UID: \"549a963a-6ae5-48c8-a7f2-2935d181e004\") " pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:21.991023 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:21.990941 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:22.295111 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:22.294973 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-678d788c65-zrb8q"] Nov 03 06:12:22.297854 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:12:22.297830 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549a963a_6ae5_48c8_a7f2_2935d181e004.slice/crio-c724d79594236f63f39f4749b5f1cab46089a77673eb27289486a32c02ea92fc WatchSource:0}: Error finding container c724d79594236f63f39f4749b5f1cab46089a77673eb27289486a32c02ea92fc: Status 404 returned error can't find the container with id c724d79594236f63f39f4749b5f1cab46089a77673eb27289486a32c02ea92fc Nov 03 06:12:22.332458 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:22.332431 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de221a90-6ca4-4606-a09e-1473cab5b01e" path="/var/lib/kubelet/pods/de221a90-6ca4-4606-a09e-1473cab5b01e/volumes" Nov 03 06:12:22.428560 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:22.428519 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-678d788c65-zrb8q" event={"ID":"549a963a-6ae5-48c8-a7f2-2935d181e004","Type":"ContainerStarted","Data":"c724d79594236f63f39f4749b5f1cab46089a77673eb27289486a32c02ea92fc"} Nov 03 06:12:23.434698 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:23.434663 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-678d788c65-zrb8q" event={"ID":"549a963a-6ae5-48c8-a7f2-2935d181e004","Type":"ContainerStarted","Data":"ee757ee12e5d18f5fecf25c6bffdaa8049387735e9911e61908e213ed58cc759"} Nov 03 06:12:23.435112 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:23.434719 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:23.485768 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:23.485713 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-678d788c65-zrb8q" podStartSLOduration=2.170071804 podStartE2EDuration="2.485698184s" totalImagesPullingTime="315.62638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:12:23.484248768 +0000 UTC m=+1031.292636036" watchObservedRunningTime="2026-11-03 06:12:23.485698184 +0000 UTC m=+1031.294085451" Nov 03 06:12:28.882745 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:12:28.882709 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:12:34.441895 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:12:34.441865 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-678d788c65-zrb8q" Nov 03 06:12:58.883613 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:12:58.883578 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:13:01.250314 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:13:01.250277 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:13:18.543811 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:13:18.543779 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:13:23.249838 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:13:23.249802 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:13:28.882850 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:13:28.882807 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:13:41.745455 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:13:41.745419 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:13:57.620290 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:13:57.620205 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:13:59.013251 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:13:59.013222 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:14:20.656384 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:14:20.656353 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:14:28.882583 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:14:28.882549 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:14:31.052132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:14:31.052096 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:14:35.629919 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:14:35.629884 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:14:55.761136 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:14:55.761101 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:14:58.886486 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:14:58.886425 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:15:00.411651 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:15:00.411619 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:15:12.487080 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:15:12.487049 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:15:12.490456 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:15:12.490434 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:15:17.397706 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:15:17.397669 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:15:21.928158 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:15:21.928120 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:15:28.882664 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:15:28.882590 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:15:40.461408 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:15:40.461357 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:15:46.065758 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:15:46.065709 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:15:58.883171 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:15:58.883131 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:16:00.723624 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:16:00.723593 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:16:11.351543 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:16:11.351505 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:16:17.923132 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:16:17.923096 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:16:28.883687 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:16:28.883649 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:16:35.927329 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:16:35.927294 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:16:41.248134 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:16:41.248096 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:16:53.060704 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:16:53.060664 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:16:58.882609 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:16:58.882520 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:17:11.153198 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:11.153162 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:17:21.635972 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:21.635933 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:17:28.883676 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:17:28.883631 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:17:40.635891 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:40.635857 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:17:46.739218 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:46.739176 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:17:50.266729 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.266689 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll"] Nov 03 06:17:50.271056 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.271036 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.286020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.285993 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-fcbaef-openshift-default-dockercfg-pcq2w\"" Nov 03 06:17:50.300905 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.300874 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll"] Nov 03 06:17:50.383204 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.383168 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-podinfo\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.383424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.383215 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5k4j\" (UniqueName: \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-kube-api-access-m5k4j\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.383424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.383279 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-certs\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.383424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.383349 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-socket\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.383591 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.383491 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-credential-socket\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.383591 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.383571 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-envoy\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.383686 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.383664 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-token\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.383743 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.383694 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istiod-ca-cert\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.383743 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.383730 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-data\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.484789 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.484744 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-credential-socket\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.484977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.484819 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-envoy\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.484977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.484866 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-token\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.484977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.484890 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istiod-ca-cert\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.484977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.484921 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-data\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.484977 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.484957 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-podinfo\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.485407 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.484996 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5k4j\" (UniqueName: \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-kube-api-access-m5k4j\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.485407 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.485039 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-certs\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.485407 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.485069 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-socket\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.485568 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.485412 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-data\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.485568 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.485426 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-socket\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.485568 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.485516 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-credential-socket\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.485568 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.485559 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-certs\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.485767 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.485692 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istiod-ca-cert\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.487290 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.487266 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-envoy\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.487582 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.487562 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-podinfo\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.510249 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.510210 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-token\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.510469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.510371 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5k4j\" (UniqueName: \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-kube-api-access-m5k4j\") pod \"e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.589349 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.589314 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:50.908187 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.908114 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll"] Nov 03 06:17:50.911221 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:17:50.911191 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f44c96_56db_4fe8_bd79_79e8a95e5e65.slice/crio-ec660566b73053ce951b0e04b057310d74c5ccd39591c3d189baa58e0c84adae WatchSource:0}: Error finding container ec660566b73053ce951b0e04b057310d74c5ccd39591c3d189baa58e0c84adae: Status 404 returned error can't find the container with id ec660566b73053ce951b0e04b057310d74c5ccd39591c3d189baa58e0c84adae Nov 03 06:17:50.913335 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.913303 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:17:50.913455 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.913377 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:17:50.913455 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:50.913437 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:17:51.075316 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:51.075278 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" event={"ID":"d7f44c96-56db-4fe8-bd79-79e8a95e5e65","Type":"ContainerStarted","Data":"6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0"} Nov 03 06:17:51.075516 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:51.075332 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" event={"ID":"d7f44c96-56db-4fe8-bd79-79e8a95e5e65","Type":"ContainerStarted","Data":"ec660566b73053ce951b0e04b057310d74c5ccd39591c3d189baa58e0c84adae"} Nov 03 06:17:51.120811 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:51.120749 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" podStartSLOduration=1.120729917 podStartE2EDuration="1.120729917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:17:51.119383998 +0000 UTC m=+1358.927771282" watchObservedRunningTime="2026-11-03 06:17:51.120729917 +0000 UTC m=+1358.929117190" Nov 03 06:17:51.590264 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:51.590229 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:51.595362 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:51.595337 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:52.080162 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:52.080139 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:52.080994 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:52.080976 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:17:54.988797 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:54.988762 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg"] Nov 03 06:17:54.992293 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:54.992274 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.008551 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.008521 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-fcbaef-serving-cert\"" Nov 03 06:17:55.022325 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.022296 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg"] Nov 03 06:17:55.031031 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.031004 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b6000d-2b3b-42c0-9769-b4d021a8905a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.031177 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.031156 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kts49\" (UniqueName: \"kubernetes.io/projected/05b6000d-2b3b-42c0-9769-b4d021a8905a-kube-api-access-kts49\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.031265 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.031248 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.031309 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.031278 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.131969 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.131921 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.131969 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.131973 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.132231 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.132094 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b6000d-2b3b-42c0-9769-b4d021a8905a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.132231 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:17:55.132200 4855 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-serving-cert: secret "maas-api-e2e-worker-single-main-fcbaef-serving-cert" not found Nov 03 06:17:55.132348 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.132255 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kts49\" (UniqueName: \"kubernetes.io/projected/05b6000d-2b3b-42c0-9769-b4d021a8905a-kube-api-access-kts49\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.132348 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:17:55.132273 4855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-maas-api-tls podName:05b6000d-2b3b-42c0-9769-b4d021a8905a nodeName:}" failed. No retries permitted until 2026-11-03 06:17:55.632253404 +0000 UTC m=+1363.440640652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-maas-api-tls") pod "maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" (UID: "05b6000d-2b3b-42c0-9769-b4d021a8905a") : secret "maas-api-e2e-worker-single-main-fcbaef-serving-cert" not found Nov 03 06:17:55.132839 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.132818 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b6000d-2b3b-42c0-9769-b4d021a8905a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.134330 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.134312 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.157036 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.157000 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kts49\" (UniqueName: \"kubernetes.io/projected/05b6000d-2b3b-42c0-9769-b4d021a8905a-kube-api-access-kts49\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.638801 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.638761 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.641158 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.641137 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:55.908947 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:55.908851 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:56.221933 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:56.221737 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg"] Nov 03 06:17:56.224531 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:17:56.224490 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b6000d_2b3b_42c0_9769_b4d021a8905a.slice/crio-3ec17ef75b96ac6c513c3afb89e447ac42b0dbcd4a17917f6f091d77b5527a6d WatchSource:0}: Error finding container 3ec17ef75b96ac6c513c3afb89e447ac42b0dbcd4a17917f6f091d77b5527a6d: Status 404 returned error can't find the container with id 3ec17ef75b96ac6c513c3afb89e447ac42b0dbcd4a17917f6f091d77b5527a6d Nov 03 06:17:56.225948 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:56.225929 4855 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 06:17:57.104636 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:57.104596 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" event={"ID":"05b6000d-2b3b-42c0-9769-b4d021a8905a","Type":"ContainerStarted","Data":"382b1fd6111c091c6ec419816e686572c230d83dec8d262780ba4864e9c93611"} Nov 03 06:17:57.104853 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:57.104647 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" event={"ID":"05b6000d-2b3b-42c0-9769-b4d021a8905a","Type":"ContainerStarted","Data":"3ec17ef75b96ac6c513c3afb89e447ac42b0dbcd4a17917f6f091d77b5527a6d"} Nov 03 06:17:57.104853 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:57.104712 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:17:57.149514 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:57.149447 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" podStartSLOduration=2.695598595 podStartE2EDuration="3.149428733s" totalImagesPullingTime="453.830138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:17:57.147132183 +0000 UTC m=+1364.955519441" watchObservedRunningTime="2026-11-03 06:17:57.149428733 +0000 UTC m=+1364.957816004" Nov 03 06:17:57.972695 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:17:57.972655 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:17:58.882951 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:17:58.882912 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:18:03.114759 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:03.114731 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:18:07.383487 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:07.382454 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": read tcp 10.133.0.2:57760->10.133.0.81:15021: read: connection reset by peer" start-of-body= Nov 03 06:18:07.383487 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:07.382529 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": read tcp 10.133.0.2:57760->10.133.0.81:15021: read: connection reset by peer" Nov 03 06:18:08.192225 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:08.192190 4855 generic.go:356] "Generic (PLEG): container finished" podID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerID="6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0" exitCode=0 Nov 03 06:18:08.192472 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:08.192268 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" event={"ID":"d7f44c96-56db-4fe8-bd79-79e8a95e5e65","Type":"ContainerDied","Data":"6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0"} Nov 03 06:18:08.192783 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:08.192761 4855 scope.go:122] "RemoveContainer" containerID="6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0" Nov 03 06:18:08.194052 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:08.194025 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:18:08.194111 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:08.194096 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:18:08.194149 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:08.194132 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:18:09.208443 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:09.208406 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" event={"ID":"d7f44c96-56db-4fe8-bd79-79e8a95e5e65","Type":"ContainerStarted","Data":"9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45"} Nov 03 06:18:09.590452 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:09.590414 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:18:09.590646 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:09.590624 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 06:18:09.590723 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:09.590702 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 06:18:10.589979 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:10.589944 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 06:18:10.590384 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:10.589994 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 06:18:11.590186 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:11.590154 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 06:18:11.590651 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:11.590220 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 06:18:12.590347 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:12.590313 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 06:18:12.590796 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:12.590371 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 06:18:13.055596 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:13.049585 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:18:13.589919 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:13.589881 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 06:18:13.590114 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:13.589935 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 06:18:14.659101 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:14.659072 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:18:14.659633 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:14.659405 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:18:14.660556 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:14.660517 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:18:28.884074 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:18:28.883987 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:18:30.174370 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:30.174336 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:18:55.078017 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:18:55.077981 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:18:58.884186 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:18:58.884137 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:19:07.024325 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.024287 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg"] Nov 03 06:19:07.024765 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.024488 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" podUID="05b6000d-2b3b-42c0-9769-b4d021a8905a" containerName="maas-api" containerID="cri-o://382b1fd6111c091c6ec419816e686572c230d83dec8d262780ba4864e9c93611" gracePeriod=30 Nov 03 06:19:07.492133 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.492102 4855 generic.go:356] "Generic (PLEG): container finished" podID="05b6000d-2b3b-42c0-9769-b4d021a8905a" containerID="382b1fd6111c091c6ec419816e686572c230d83dec8d262780ba4864e9c93611" exitCode=0 Nov 03 06:19:07.492133 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.492140 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" event={"ID":"05b6000d-2b3b-42c0-9769-b4d021a8905a","Type":"ContainerDied","Data":"382b1fd6111c091c6ec419816e686572c230d83dec8d262780ba4864e9c93611"} Nov 03 06:19:07.520111 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.520087 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:19:07.612826 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.612755 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05b6000d-2b3b-42c0-9769-b4d021a8905a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b6000d-2b3b-42c0-9769-b4d021a8905a-openshift-service-ca\") pod \"05b6000d-2b3b-42c0-9769-b4d021a8905a\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " Nov 03 06:19:07.612986 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.612856 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b6000d-2b3b-42c0-9769-b4d021a8905a-kube-api-access-kts49\" (UniqueName: \"kubernetes.io/projected/05b6000d-2b3b-42c0-9769-b4d021a8905a-kube-api-access-kts49\") pod \"05b6000d-2b3b-42c0-9769-b4d021a8905a\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " Nov 03 06:19:07.612986 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.612928 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-metrics-certs\") pod \"05b6000d-2b3b-42c0-9769-b4d021a8905a\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " Nov 03 06:19:07.613130 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.613035 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-maas-api-tls\") pod \"05b6000d-2b3b-42c0-9769-b4d021a8905a\" (UID: \"05b6000d-2b3b-42c0-9769-b4d021a8905a\") " Nov 03 06:19:07.613130 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.613090 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b6000d-2b3b-42c0-9769-b4d021a8905a-openshift-service-ca" pod "05b6000d-2b3b-42c0-9769-b4d021a8905a" (UID: "05b6000d-2b3b-42c0-9769-b4d021a8905a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:19:07.613325 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.613308 4855 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b6000d-2b3b-42c0-9769-b4d021a8905a-openshift-service-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:07.615010 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.614977 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b6000d-2b3b-42c0-9769-b4d021a8905a-kube-api-access-kts49" pod "05b6000d-2b3b-42c0-9769-b4d021a8905a" (UID: "05b6000d-2b3b-42c0-9769-b4d021a8905a"). InnerVolumeSpecName "kube-api-access-kts49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:19:07.615110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.615011 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-metrics-certs" pod "05b6000d-2b3b-42c0-9769-b4d021a8905a" (UID: "05b6000d-2b3b-42c0-9769-b4d021a8905a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:19:07.615110 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.615042 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-maas-api-tls" pod "05b6000d-2b3b-42c0-9769-b4d021a8905a" (UID: "05b6000d-2b3b-42c0-9769-b4d021a8905a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:19:07.713894 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.713865 4855 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-maas-api-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:07.713894 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.713891 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kts49\" (UniqueName: \"kubernetes.io/projected/05b6000d-2b3b-42c0-9769-b4d021a8905a-kube-api-access-kts49\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:07.714090 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:07.713906 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b6000d-2b3b-42c0-9769-b4d021a8905a-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:08.501929 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:08.501891 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" event={"ID":"05b6000d-2b3b-42c0-9769-b4d021a8905a","Type":"ContainerDied","Data":"3ec17ef75b96ac6c513c3afb89e447ac42b0dbcd4a17917f6f091d77b5527a6d"} Nov 03 06:19:08.502337 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:08.501905 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg" Nov 03 06:19:08.502337 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:08.501947 4855 scope.go:122] "RemoveContainer" containerID="382b1fd6111c091c6ec419816e686572c230d83dec8d262780ba4864e9c93611" Nov 03 06:19:08.555352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:08.555320 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg"] Nov 03 06:19:08.567064 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:08.567039 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fcbaef-86686f8959-j9pxg"] Nov 03 06:19:09.750729 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:09.750683 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:19:10.332635 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:10.332599 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b6000d-2b3b-42c0-9769-b4d021a8905a" path="/var/lib/kubelet/pods/05b6000d-2b3b-42c0-9769-b4d021a8905a/volumes" Nov 03 06:19:15.250013 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:15.249977 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll"] Nov 03 06:19:15.250441 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:15.250200 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" containerID="cri-o://9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45" gracePeriod=30 Nov 03 06:19:20.503491 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.503461 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:19:20.558286 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.558251 4855 generic.go:356] "Generic (PLEG): container finished" podID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerID="9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45" exitCode=0 Nov 03 06:19:20.558475 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.558298 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" Nov 03 06:19:20.558475 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.558307 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" event={"ID":"d7f44c96-56db-4fe8-bd79-79e8a95e5e65","Type":"ContainerDied","Data":"9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45"} Nov 03 06:19:20.558475 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.558355 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll" event={"ID":"d7f44c96-56db-4fe8-bd79-79e8a95e5e65","Type":"ContainerDied","Data":"ec660566b73053ce951b0e04b057310d74c5ccd39591c3d189baa58e0c84adae"} Nov 03 06:19:20.558475 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.558377 4855 scope.go:122] "RemoveContainer" containerID="9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45" Nov 03 06:19:20.568166 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.568139 4855 scope.go:122] "RemoveContainer" containerID="6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0" Nov 03 06:19:20.576046 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.576028 4855 scope.go:122] "RemoveContainer" containerID="9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45" Nov 03 06:19:20.576353 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:19:20.576324 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45\": container with ID starting with 9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45 not found: ID does not exist" containerID="9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45" Nov 03 06:19:20.576436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.576362 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45"} err="failed to get container status \"9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45\": rpc error: code = NotFound desc = could not find container \"9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45\": container with ID starting with 9416e22ab461e985c4ac4579ef059047274f224e4f0fa406dd15ae36411fbc45 not found: ID does not exist" Nov 03 06:19:20.576436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.576379 4855 scope.go:122] "RemoveContainer" containerID="6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0" Nov 03 06:19:20.576704 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:19:20.576679 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0\": container with ID starting with 6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0 not found: ID does not exist" containerID="6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0" Nov 03 06:19:20.576772 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.576715 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0"} err="failed to get container status \"6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0\": rpc error: code = NotFound desc = could not find container \"6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0\": container with ID starting with 6b3c7611239cd5f4fb75fc47473d60847e3f5eb4d9cbd4a5ca3ed93a2da2fac0 not found: ID does not exist" Nov 03 06:19:20.658425 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658383 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-token\" (UniqueName: \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-token\") pod \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " Nov 03 06:19:20.658586 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658436 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-envoy\") pod \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " Nov 03 06:19:20.658586 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658465 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-socket\") pod \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " Nov 03 06:19:20.658586 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658496 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-kube-api-access-m5k4j\" (UniqueName: \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-kube-api-access-m5k4j\") pod \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " Nov 03 06:19:20.658586 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658566 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-certs\") pod \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " Nov 03 06:19:20.658793 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658708 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-data\") pod \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " Nov 03 06:19:20.658793 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658763 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-podinfo\") pod \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " Nov 03 06:19:20.658896 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658829 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istiod-ca-cert\") pod \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " Nov 03 06:19:20.658955 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658763 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-socket" pod "d7f44c96-56db-4fe8-bd79-79e8a95e5e65" (UID: "d7f44c96-56db-4fe8-bd79-79e8a95e5e65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:19:20.658955 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658868 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-certs" pod "d7f44c96-56db-4fe8-bd79-79e8a95e5e65" (UID: "d7f44c96-56db-4fe8-bd79-79e8a95e5e65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:19:20.658955 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658940 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-data" pod "d7f44c96-56db-4fe8-bd79-79e8a95e5e65" (UID: "d7f44c96-56db-4fe8-bd79-79e8a95e5e65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:19:20.659097 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.658951 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-credential-socket\") pod \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\" (UID: \"d7f44c96-56db-4fe8-bd79-79e8a95e5e65\") " Nov 03 06:19:20.659238 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.659214 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-credential-socket" pod "d7f44c96-56db-4fe8-bd79-79e8a95e5e65" (UID: "d7f44c96-56db-4fe8-bd79-79e8a95e5e65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:19:20.659322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.659230 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istiod-ca-cert" pod "d7f44c96-56db-4fe8-bd79-79e8a95e5e65" (UID: "d7f44c96-56db-4fe8-bd79-79e8a95e5e65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:19:20.659322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.659268 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:20.659322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.659288 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:20.659322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.659302 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:20.660686 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.660665 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-token" pod "d7f44c96-56db-4fe8-bd79-79e8a95e5e65" (UID: "d7f44c96-56db-4fe8-bd79-79e8a95e5e65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:19:20.660782 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.660736 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-kube-api-access-m5k4j" pod "d7f44c96-56db-4fe8-bd79-79e8a95e5e65" (UID: "d7f44c96-56db-4fe8-bd79-79e8a95e5e65"). InnerVolumeSpecName "kube-api-access-m5k4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:19:20.660964 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.660946 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-envoy" pod "d7f44c96-56db-4fe8-bd79-79e8a95e5e65" (UID: "d7f44c96-56db-4fe8-bd79-79e8a95e5e65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:19:20.661090 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.661074 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-podinfo" pod "d7f44c96-56db-4fe8-bd79-79e8a95e5e65" (UID: "d7f44c96-56db-4fe8-bd79-79e8a95e5e65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:19:20.759807 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.759774 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:20.759807 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.759802 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:20.759807 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.759811 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5k4j\" (UniqueName: \"kubernetes.io/projected/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-kube-api-access-m5k4j\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:20.760027 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.759820 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:20.760027 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.759830 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:20.760027 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.759837 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f44c96-56db-4fe8-bd79-79e8a95e5e65-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:19:20.903327 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.903300 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll"] Nov 03 06:19:20.915492 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:20.915465 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fcbaef-openshift-default-684dbf67c7dgkll"] Nov 03 06:19:22.331703 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:22.331666 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" path="/var/lib/kubelet/pods/d7f44c96-56db-4fe8-bd79-79e8a95e5e65/volumes" Nov 03 06:19:28.731824 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:28.731789 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:19:28.883399 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:19:28.883349 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:19:34.926215 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:19:34.926178 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:19:58.882915 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:19:58.882839 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:20:12.514595 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:20:12.514559 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:20:12.518600 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:20:12.518577 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:20:28.883948 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:20:28.883910 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:20:58.883951 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:20:58.883909 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:21:02.647647 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:02.647610 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:21:17.446884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:17.446849 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:21:22.253516 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:22.253474 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-nkx8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:21:22.254150 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:22.254115 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-nkx8k" podUID="fd2713fe-2336-4f1e-8013-f8ede390211d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:21:22.858853 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:22.858816 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:21:28.883684 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:21:28.883609 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:21:38.669003 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:38.668967 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:21:39.607569 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.607538 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx"] Nov 03 06:21:39.608030 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.608011 4855 cpu_manager.go:414] "containerMap: removing container" podUID="05b6000d-2b3b-42c0-9769-b4d021a8905a" containerName="maas-api" Nov 03 06:21:39.608030 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.608030 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b6000d-2b3b-42c0-9769-b4d021a8905a" containerName="maas-api" Nov 03 06:21:39.608180 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.608048 4855 cpu_manager.go:414] "containerMap: removing container" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" Nov 03 06:21:39.608180 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.608053 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" Nov 03 06:21:39.608180 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.608063 4855 cpu_manager.go:414] "containerMap: removing container" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" Nov 03 06:21:39.608180 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.608068 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" Nov 03 06:21:39.608180 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.608128 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" Nov 03 06:21:39.608180 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.608139 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b6000d-2b3b-42c0-9769-b4d021a8905a" containerName="maas-api" Nov 03 06:21:39.611202 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.611184 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.625933 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.625913 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-7c2bc6-openshift-default-dockercfg-9dktx\"" Nov 03 06:21:39.639970 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.639948 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx"] Nov 03 06:21:39.657890 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.657859 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-data\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.658013 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.657919 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c952cfa1-3312-4605-8557-a8f6cc85dcda-istiod-ca-cert\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.658080 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.658067 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-socket\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.658125 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.658088 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-token\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.658125 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.658112 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wtmb\" (UniqueName: \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-kube-api-access-4wtmb\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.658206 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.658134 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-certs\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.658267 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.658247 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-envoy\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.658308 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.658288 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-credential-socket\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.658366 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.658351 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-podinfo\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.759695 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.759657 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-podinfo\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.759717 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-data\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.759760 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c952cfa1-3312-4605-8557-a8f6cc85dcda-istiod-ca-cert\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.759853 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-socket\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.759878 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-token\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.759913 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wtmb\" (UniqueName: \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-kube-api-access-4wtmb\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.759947 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-certs\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.759985 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-envoy\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760196 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.760028 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-credential-socket\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.760225 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-data\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.760337 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-credential-socket\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.760526 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-certs\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760639 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.760606 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-socket\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.760765 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.760676 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c952cfa1-3312-4605-8557-a8f6cc85dcda-istiod-ca-cert\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.762336 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.762309 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-envoy\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.762650 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.762634 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-podinfo\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.783980 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.783954 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-token\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.784879 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.784858 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wtmb\" (UniqueName: \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-kube-api-access-4wtmb\") pod \"e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:39.925237 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:39.925150 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:40.244645 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:40.244618 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx"] Nov 03 06:21:40.247481 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:21:40.247453 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc952cfa1_3312_4605_8557_a8f6cc85dcda.slice/crio-bbecfd801da0a135e4e02aa7437dc6e6770ff45f9b247eb600b3267410384120 WatchSource:0}: Error finding container bbecfd801da0a135e4e02aa7437dc6e6770ff45f9b247eb600b3267410384120: Status 404 returned error can't find the container with id bbecfd801da0a135e4e02aa7437dc6e6770ff45f9b247eb600b3267410384120 Nov 03 06:21:40.249722 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:40.249689 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:21:40.249803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:40.249758 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:21:40.249803 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:40.249787 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:21:40.256128 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:40.256098 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" event={"ID":"c952cfa1-3312-4605-8557-a8f6cc85dcda","Type":"ContainerStarted","Data":"bbecfd801da0a135e4e02aa7437dc6e6770ff45f9b247eb600b3267410384120"} Nov 03 06:21:41.261691 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:41.261648 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" event={"ID":"c952cfa1-3312-4605-8557-a8f6cc85dcda","Type":"ContainerStarted","Data":"19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e"} Nov 03 06:21:41.331216 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:41.331162 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" podStartSLOduration=2.331149727 podStartE2EDuration="2.331149727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:21:41.318603796 +0000 UTC m=+1589.126991065" watchObservedRunningTime="2026-11-03 06:21:41.331149727 +0000 UTC m=+1589.139536985" Nov 03 06:21:41.925726 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:41.925681 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:41.930616 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:41.930591 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:42.265989 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:42.265907 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:42.266790 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:42.266769 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:21:45.309004 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.308968 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs"] Nov 03 06:21:45.309461 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.309445 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f44c96-56db-4fe8-bd79-79e8a95e5e65" containerName="istio-proxy" Nov 03 06:21:45.312697 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.312673 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.348491 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.348458 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-7c2bc6-serving-cert\"" Nov 03 06:21:45.351718 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.351677 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs"] Nov 03 06:21:45.423183 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.423152 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.423183 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.423187 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.423409 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.423320 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5spg5\" (UniqueName: \"kubernetes.io/projected/f1848de4-7358-4400-8b1d-f612094b7d3c-kube-api-access-5spg5\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.423469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.423452 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1848de4-7358-4400-8b1d-f612094b7d3c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.524135 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.524086 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5spg5\" (UniqueName: \"kubernetes.io/projected/f1848de4-7358-4400-8b1d-f612094b7d3c-kube-api-access-5spg5\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.524307 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.524197 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1848de4-7358-4400-8b1d-f612094b7d3c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.524307 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.524243 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.524307 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.524270 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.524868 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.524841 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1848de4-7358-4400-8b1d-f612094b7d3c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.526758 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.526733 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.527006 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.526983 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.547753 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.547723 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5spg5\" (UniqueName: \"kubernetes.io/projected/f1848de4-7358-4400-8b1d-f612094b7d3c-kube-api-access-5spg5\") pod \"maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.626429 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.626330 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:45.928109 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:45.928076 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs"] Nov 03 06:21:45.930839 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:21:45.930810 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1848de4_7358_4400_8b1d_f612094b7d3c.slice/crio-1aee3ec3e485e5a9af19fe320128a8b36a0a6f29f450610e0ffba158fbe014b9 WatchSource:0}: Error finding container 1aee3ec3e485e5a9af19fe320128a8b36a0a6f29f450610e0ffba158fbe014b9: Status 404 returned error can't find the container with id 1aee3ec3e485e5a9af19fe320128a8b36a0a6f29f450610e0ffba158fbe014b9 Nov 03 06:21:46.292813 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:46.292713 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" event={"ID":"f1848de4-7358-4400-8b1d-f612094b7d3c","Type":"ContainerStarted","Data":"1aee3ec3e485e5a9af19fe320128a8b36a0a6f29f450610e0ffba158fbe014b9"} Nov 03 06:21:47.298495 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:47.298453 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" event={"ID":"f1848de4-7358-4400-8b1d-f612094b7d3c","Type":"ContainerStarted","Data":"3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de"} Nov 03 06:21:47.298997 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:47.298561 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:47.352551 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:47.352492 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" podStartSLOduration=1.931284062 podStartE2EDuration="2.352478661s" totalImagesPullingTime="421.194599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:21:47.351189031 +0000 UTC m=+1595.159576314" watchObservedRunningTime="2026-11-03 06:21:47.352478661 +0000 UTC m=+1595.160865929" Nov 03 06:21:48.978415 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:48.978348 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:21:53.308832 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:53.308802 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:21:58.307566 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:58.307518 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:21:58.308054 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:58.307599 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:21:58.883642 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:21:58.883567 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:21:59.369711 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:59.369675 4855 generic.go:356] "Generic (PLEG): container finished" podID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerID="19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e" exitCode=0 Nov 03 06:21:59.370214 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:59.369733 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" event={"ID":"c952cfa1-3312-4605-8557-a8f6cc85dcda","Type":"ContainerDied","Data":"19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e"} Nov 03 06:21:59.370214 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:59.370128 4855 scope.go:122] "RemoveContainer" containerID="19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e" Nov 03 06:21:59.371312 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:59.371284 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:21:59.371411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:59.371347 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:21:59.371411 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:21:59.371377 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:22:00.380030 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:00.379986 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" event={"ID":"c952cfa1-3312-4605-8557-a8f6cc85dcda","Type":"ContainerStarted","Data":"f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac"} Nov 03 06:22:00.926280 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:00.926237 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:22:00.926531 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:00.926505 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 06:22:00.926614 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:00.926559 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 06:22:01.926544 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:01.926501 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 06:22:01.926951 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:01.926571 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 06:22:02.925692 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:02.925659 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 06:22:02.925860 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:02.925714 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 06:22:03.616676 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:03.616643 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:22:03.925771 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:03.925686 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 06:22:03.925771 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:03.925734 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 06:22:04.926382 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:04.926351 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 06:22:04.926825 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:04.926443 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 06:22:05.929314 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:05.929285 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:22:05.929744 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:05.929628 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:22:05.930420 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:05.930375 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:22:20.253347 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:20.253310 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:22:28.883895 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:22:28.883849 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:22:44.752262 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:44.752230 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:22:56.254612 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:56.254529 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs"] Nov 03 06:22:56.255072 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:56.254711 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" podUID="f1848de4-7358-4400-8b1d-f612094b7d3c" containerName="maas-api" containerID="cri-o://3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de" gracePeriod=30 Nov 03 06:22:57.508957 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.508919 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:22:57.591561 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.591481 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1848de4-7358-4400-8b1d-f612094b7d3c-kube-api-access-5spg5\" (UniqueName: \"kubernetes.io/projected/f1848de4-7358-4400-8b1d-f612094b7d3c-kube-api-access-5spg5\") pod \"f1848de4-7358-4400-8b1d-f612094b7d3c\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " Nov 03 06:22:57.591561 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.591554 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-metrics-certs\") pod \"f1848de4-7358-4400-8b1d-f612094b7d3c\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " Nov 03 06:22:57.591746 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.591662 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-maas-api-tls\") pod \"f1848de4-7358-4400-8b1d-f612094b7d3c\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " Nov 03 06:22:57.591746 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.591685 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1848de4-7358-4400-8b1d-f612094b7d3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1848de4-7358-4400-8b1d-f612094b7d3c-openshift-service-ca\") pod \"f1848de4-7358-4400-8b1d-f612094b7d3c\" (UID: \"f1848de4-7358-4400-8b1d-f612094b7d3c\") " Nov 03 06:22:57.592095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.592052 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1848de4-7358-4400-8b1d-f612094b7d3c-openshift-service-ca" pod "f1848de4-7358-4400-8b1d-f612094b7d3c" (UID: "f1848de4-7358-4400-8b1d-f612094b7d3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:22:57.593614 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.593583 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1848de4-7358-4400-8b1d-f612094b7d3c-kube-api-access-5spg5" pod "f1848de4-7358-4400-8b1d-f612094b7d3c" (UID: "f1848de4-7358-4400-8b1d-f612094b7d3c"). InnerVolumeSpecName "kube-api-access-5spg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:22:57.593702 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.593681 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-maas-api-tls" pod "f1848de4-7358-4400-8b1d-f612094b7d3c" (UID: "f1848de4-7358-4400-8b1d-f612094b7d3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:22:57.593747 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.593696 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-metrics-certs" pod "f1848de4-7358-4400-8b1d-f612094b7d3c" (UID: "f1848de4-7358-4400-8b1d-f612094b7d3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:22:57.673903 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.673868 4855 generic.go:356] "Generic (PLEG): container finished" podID="f1848de4-7358-4400-8b1d-f612094b7d3c" containerID="3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de" exitCode=0 Nov 03 06:22:57.674059 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.673919 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" Nov 03 06:22:57.674059 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.673930 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" event={"ID":"f1848de4-7358-4400-8b1d-f612094b7d3c","Type":"ContainerDied","Data":"3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de"} Nov 03 06:22:57.674059 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.673971 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs" event={"ID":"f1848de4-7358-4400-8b1d-f612094b7d3c","Type":"ContainerDied","Data":"1aee3ec3e485e5a9af19fe320128a8b36a0a6f29f450610e0ffba158fbe014b9"} Nov 03 06:22:57.674059 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.673991 4855 scope.go:122] "RemoveContainer" containerID="3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de" Nov 03 06:22:57.683229 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.683211 4855 scope.go:122] "RemoveContainer" containerID="3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de" Nov 03 06:22:57.683548 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:22:57.683524 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de\": container with ID starting with 3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de not found: ID does not exist" containerID="3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de" Nov 03 06:22:57.683622 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.683557 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de"} err="failed to get container status \"3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de\": rpc error: code = NotFound desc = could not find container \"3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de\": container with ID starting with 3d2900c76b885185066ffd15e9d3e0628b34478952b943a238739345be61a1de not found: ID does not exist" Nov 03 06:22:57.693537 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.693517 4855 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-maas-api-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:22:57.693593 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.693544 4855 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1848de4-7358-4400-8b1d-f612094b7d3c-openshift-service-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:22:57.693593 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.693560 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5spg5\" (UniqueName: \"kubernetes.io/projected/f1848de4-7358-4400-8b1d-f612094b7d3c-kube-api-access-5spg5\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:22:57.693593 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.693575 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1848de4-7358-4400-8b1d-f612094b7d3c-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:22:57.727636 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.727611 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs"] Nov 03 06:22:57.732010 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:57.731988 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7c2bc6-7bf5d7dfcc-rnsbs"] Nov 03 06:22:58.332402 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:58.332363 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1848de4-7358-4400-8b1d-f612094b7d3c" path="/var/lib/kubelet/pods/f1848de4-7358-4400-8b1d-f612094b7d3c/volumes" Nov 03 06:22:58.707920 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:22:58.707837 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:22:58.883253 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:22:58.883213 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:23:03.850749 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:03.850715 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx"] Nov 03 06:23:03.851354 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:03.850967 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" containerID="cri-o://f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac" gracePeriod=30 Nov 03 06:23:05.931565 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:05.931522 4855 patch_prober.go:29] interesting pod/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 06:23:05.931983 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:05.931596 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:23:09.105910 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.105885 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:23:09.206858 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.206779 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-kube-api-access-4wtmb\" (UniqueName: \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-kube-api-access-4wtmb\") pod \"c952cfa1-3312-4605-8557-a8f6cc85dcda\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " Nov 03 06:23:09.206858 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.206820 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c952cfa1-3312-4605-8557-a8f6cc85dcda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c952cfa1-3312-4605-8557-a8f6cc85dcda-istiod-ca-cert\") pod \"c952cfa1-3312-4605-8557-a8f6cc85dcda\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " Nov 03 06:23:09.206858 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.206851 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-token\" (UniqueName: \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-token\") pod \"c952cfa1-3312-4605-8557-a8f6cc85dcda\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " Nov 03 06:23:09.207136 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.206882 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-podinfo\") pod \"c952cfa1-3312-4605-8557-a8f6cc85dcda\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " Nov 03 06:23:09.207136 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.206967 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-data\") pod \"c952cfa1-3312-4605-8557-a8f6cc85dcda\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " Nov 03 06:23:09.207136 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207063 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-certs\") pod \"c952cfa1-3312-4605-8557-a8f6cc85dcda\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " Nov 03 06:23:09.207136 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207095 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-envoy\") pod \"c952cfa1-3312-4605-8557-a8f6cc85dcda\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " Nov 03 06:23:09.207136 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207135 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-socket\") pod \"c952cfa1-3312-4605-8557-a8f6cc85dcda\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " Nov 03 06:23:09.207432 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207166 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-credential-socket\") pod \"c952cfa1-3312-4605-8557-a8f6cc85dcda\" (UID: \"c952cfa1-3312-4605-8557-a8f6cc85dcda\") " Nov 03 06:23:09.207432 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207279 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-data" pod "c952cfa1-3312-4605-8557-a8f6cc85dcda" (UID: "c952cfa1-3312-4605-8557-a8f6cc85dcda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:23:09.207432 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207320 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-certs" pod "c952cfa1-3312-4605-8557-a8f6cc85dcda" (UID: "c952cfa1-3312-4605-8557-a8f6cc85dcda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:23:09.207432 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207364 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c952cfa1-3312-4605-8557-a8f6cc85dcda-istiod-ca-cert" pod "c952cfa1-3312-4605-8557-a8f6cc85dcda" (UID: "c952cfa1-3312-4605-8557-a8f6cc85dcda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:23:09.207652 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207549 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-socket" pod "c952cfa1-3312-4605-8557-a8f6cc85dcda" (UID: "c952cfa1-3312-4605-8557-a8f6cc85dcda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:23:09.207652 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207571 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-credential-socket" pod "c952cfa1-3312-4605-8557-a8f6cc85dcda" (UID: "c952cfa1-3312-4605-8557-a8f6cc85dcda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:23:09.207724 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207699 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c952cfa1-3312-4605-8557-a8f6cc85dcda-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:23:09.207724 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207717 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:23:09.207816 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207731 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:23:09.207816 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207743 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:23:09.207816 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.207757 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:23:09.209118 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.209097 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-token" pod "c952cfa1-3312-4605-8557-a8f6cc85dcda" (UID: "c952cfa1-3312-4605-8557-a8f6cc85dcda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:23:09.209216 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.209112 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-podinfo" pod "c952cfa1-3312-4605-8557-a8f6cc85dcda" (UID: "c952cfa1-3312-4605-8557-a8f6cc85dcda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:23:09.209709 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.209691 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-kube-api-access-4wtmb" pod "c952cfa1-3312-4605-8557-a8f6cc85dcda" (UID: "c952cfa1-3312-4605-8557-a8f6cc85dcda"). InnerVolumeSpecName "kube-api-access-4wtmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:23:09.209789 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.209691 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-envoy" pod "c952cfa1-3312-4605-8557-a8f6cc85dcda" (UID: "c952cfa1-3312-4605-8557-a8f6cc85dcda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:23:09.308331 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.308298 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wtmb\" (UniqueName: \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-kube-api-access-4wtmb\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:23:09.308331 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.308326 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:23:09.308331 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.308336 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:23:09.308570 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.308345 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c952cfa1-3312-4605-8557-a8f6cc85dcda-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:23:09.733797 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.733768 4855 generic.go:356] "Generic (PLEG): container finished" podID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerID="f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac" exitCode=0 Nov 03 06:23:09.733972 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.733815 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" Nov 03 06:23:09.733972 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.733848 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" event={"ID":"c952cfa1-3312-4605-8557-a8f6cc85dcda","Type":"ContainerDied","Data":"f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac"} Nov 03 06:23:09.733972 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.733892 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx" event={"ID":"c952cfa1-3312-4605-8557-a8f6cc85dcda","Type":"ContainerDied","Data":"bbecfd801da0a135e4e02aa7437dc6e6770ff45f9b247eb600b3267410384120"} Nov 03 06:23:09.733972 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.733917 4855 scope.go:122] "RemoveContainer" containerID="f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac" Nov 03 06:23:09.743314 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.743294 4855 scope.go:122] "RemoveContainer" containerID="19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e" Nov 03 06:23:09.751079 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.751065 4855 scope.go:122] "RemoveContainer" containerID="f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac" Nov 03 06:23:09.751318 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:23:09.751302 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac\": container with ID starting with f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac not found: ID does not exist" containerID="f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac" Nov 03 06:23:09.751368 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.751325 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac"} err="failed to get container status \"f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac\": rpc error: code = NotFound desc = could not find container \"f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac\": container with ID starting with f13db1b16c31cbd69396f25bda7f633b75d785ceec5aa96a1e1195b8547ffaac not found: ID does not exist" Nov 03 06:23:09.751368 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.751343 4855 scope.go:122] "RemoveContainer" containerID="19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e" Nov 03 06:23:09.751578 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:23:09.751561 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e\": container with ID starting with 19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e not found: ID does not exist" containerID="19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e" Nov 03 06:23:09.751621 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.751585 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e"} err="failed to get container status \"19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e\": rpc error: code = NotFound desc = could not find container \"19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e\": container with ID starting with 19d04a1e5432823e645798ace55cadd4b9e0fb749e4e582a55566cef60b4c57e not found: ID does not exist" Nov 03 06:23:09.802331 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.802309 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx"] Nov 03 06:23:09.815831 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:09.815809 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7c2bc6-openshift-default-549dc48c495g6xx"] Nov 03 06:23:10.332038 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:10.332007 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" path="/var/lib/kubelet/pods/c952cfa1-3312-4605-8557-a8f6cc85dcda/volumes" Nov 03 06:23:10.636091 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:10.635998 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:23:24.772214 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.772180 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx"] Nov 03 06:23:24.772712 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.772634 4855 cpu_manager.go:414] "containerMap: removing container" podUID="f1848de4-7358-4400-8b1d-f612094b7d3c" containerName="maas-api" Nov 03 06:23:24.772712 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.772652 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1848de4-7358-4400-8b1d-f612094b7d3c" containerName="maas-api" Nov 03 06:23:24.772712 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.772678 4855 cpu_manager.go:414] "containerMap: removing container" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" Nov 03 06:23:24.772712 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.772684 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" Nov 03 06:23:24.772883 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.772773 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" Nov 03 06:23:24.772883 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.772785 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" Nov 03 06:23:24.772883 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.772795 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1848de4-7358-4400-8b1d-f612094b7d3c" containerName="maas-api" Nov 03 06:23:24.777469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.777449 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.807548 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.807517 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6d432a37-kserve-self-signed-certs\"" Nov 03 06:23:24.809097 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.809076 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx"] Nov 03 06:23:24.809868 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.809848 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-dshm\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.809927 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.809884 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-tmp-dir\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.809927 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.809915 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-home\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.810080 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.810062 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-model-cache\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.810148 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.810130 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6275a64e-ec9f-408a-a234-37abfae3ec71-tls-certs\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.810231 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.810208 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjstm\" (UniqueName: \"kubernetes.io/projected/6275a64e-ec9f-408a-a234-37abfae3ec71-kube-api-access-cjstm\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.910899 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.910856 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-model-cache\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.911056 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.910913 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6275a64e-ec9f-408a-a234-37abfae3ec71-tls-certs\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.911056 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.910950 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjstm\" (UniqueName: \"kubernetes.io/projected/6275a64e-ec9f-408a-a234-37abfae3ec71-kube-api-access-cjstm\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.911056 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.911018 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-dshm\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.911056 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.911052 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-tmp-dir\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.911284 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.911094 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-home\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.911376 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.911353 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-model-cache\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.911478 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.911424 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-home\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.911555 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.911518 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-tmp-dir\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.913343 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.913315 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-dshm\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.913527 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.913510 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6275a64e-ec9f-408a-a234-37abfae3ec71-tls-certs\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:24.935592 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:24.935567 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjstm\" (UniqueName: \"kubernetes.io/projected/6275a64e-ec9f-408a-a234-37abfae3ec71-kube-api-access-cjstm\") pod \"e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:25.092438 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:25.092402 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:25.396808 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:25.396784 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx"] Nov 03 06:23:25.399139 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:23:25.399110 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6275a64e_ec9f_408a_a234_37abfae3ec71.slice/crio-696ae1dfdbd1829a882d30fb6c1ab054498eb1ccd0bed7b641fa549836dbc800 WatchSource:0}: Error finding container 696ae1dfdbd1829a882d30fb6c1ab054498eb1ccd0bed7b641fa549836dbc800: Status 404 returned error can't find the container with id 696ae1dfdbd1829a882d30fb6c1ab054498eb1ccd0bed7b641fa549836dbc800 Nov 03 06:23:25.830586 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:25.830552 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" event={"ID":"6275a64e-ec9f-408a-a234-37abfae3ec71","Type":"ContainerStarted","Data":"a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83"} Nov 03 06:23:25.830586 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:25.830590 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" event={"ID":"6275a64e-ec9f-408a-a234-37abfae3ec71","Type":"ContainerStarted","Data":"696ae1dfdbd1829a882d30fb6c1ab054498eb1ccd0bed7b641fa549836dbc800"} Nov 03 06:23:25.875264 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:25.875213 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" podStartSLOduration=1.875196706 podStartE2EDuration="1.875196706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:23:25.873687271 +0000 UTC m=+1693.682074538" watchObservedRunningTime="2026-11-03 06:23:25.875196706 +0000 UTC m=+1693.683583975" Nov 03 06:23:26.093130 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:26.093050 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:27.162122 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:27.162086 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:23:28.929455 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:23:28.929420 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:23:35.093384 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:35.093349 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:35.106158 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:35.106123 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:35.949356 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:35.949331 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:23:48.612540 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:23:48.612510 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:23:58.883027 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:23:58.882988 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:24:02.816055 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:02.816014 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:24:02.869434 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:02.869399 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx"] Nov 03 06:24:05.795414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:05.791497 4855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ssx5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 06:24:05.795414 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:05.791581 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ssx5p" podUID="17697121-d871-43b4-b1e8-5e74fb513cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 06:24:09.201992 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:09.201956 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:24:17.889563 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:17.889507 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" podUID="6275a64e-ec9f-408a-a234-37abfae3ec71" containerName="main" containerID="cri-o://a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83" gracePeriod=45 Nov 03 06:24:17.938200 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:17.938161 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" podUID="6275a64e-ec9f-408a-a234-37abfae3ec71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": dial tcp 10.133.0.85:8000: connect: connection refused" Nov 03 06:24:17.964765 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:17.964737 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:24:18.148087 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.148028 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:24:18.154121 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.154098 4855 generic.go:356] "Generic (PLEG): container finished" podID="6275a64e-ec9f-408a-a234-37abfae3ec71" containerID="a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83" exitCode=0 Nov 03 06:24:18.154222 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.154131 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" Nov 03 06:24:18.154222 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.154153 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" event={"ID":"6275a64e-ec9f-408a-a234-37abfae3ec71","Type":"ContainerDied","Data":"a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83"} Nov 03 06:24:18.154222 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.154193 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx" event={"ID":"6275a64e-ec9f-408a-a234-37abfae3ec71","Type":"ContainerDied","Data":"696ae1dfdbd1829a882d30fb6c1ab054498eb1ccd0bed7b641fa549836dbc800"} Nov 03 06:24:18.154222 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.154213 4855 scope.go:122] "RemoveContainer" containerID="a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83" Nov 03 06:24:18.164371 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.164355 4855 scope.go:122] "RemoveContainer" containerID="a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83" Nov 03 06:24:18.164691 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:24:18.164673 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83\": container with ID starting with a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83 not found: ID does not exist" containerID="a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83" Nov 03 06:24:18.164762 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.164700 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83"} err="failed to get container status \"a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83\": rpc error: code = NotFound desc = could not find container \"a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83\": container with ID starting with a444040918e80ae4f7512a6b116bd28c042dce103b9f5c37ea69b25d813d8b83 not found: ID does not exist" Nov 03 06:24:18.264879 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.264846 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6275a64e-ec9f-408a-a234-37abfae3ec71-tls-certs\" (UniqueName: \"kubernetes.io/secret/6275a64e-ec9f-408a-a234-37abfae3ec71-tls-certs\") pod \"6275a64e-ec9f-408a-a234-37abfae3ec71\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " Nov 03 06:24:18.265042 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.264916 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-tmp-dir\") pod \"6275a64e-ec9f-408a-a234-37abfae3ec71\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " Nov 03 06:24:18.265042 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.264980 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-home\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-home\") pod \"6275a64e-ec9f-408a-a234-37abfae3ec71\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " Nov 03 06:24:18.265042 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.265016 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-dshm\") pod \"6275a64e-ec9f-408a-a234-37abfae3ec71\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " Nov 03 06:24:18.265166 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.265102 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-model-cache\") pod \"6275a64e-ec9f-408a-a234-37abfae3ec71\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " Nov 03 06:24:18.265166 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.265138 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6275a64e-ec9f-408a-a234-37abfae3ec71-kube-api-access-cjstm\" (UniqueName: \"kubernetes.io/projected/6275a64e-ec9f-408a-a234-37abfae3ec71-kube-api-access-cjstm\") pod \"6275a64e-ec9f-408a-a234-37abfae3ec71\" (UID: \"6275a64e-ec9f-408a-a234-37abfae3ec71\") " Nov 03 06:24:18.265253 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.265209 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-tmp-dir" pod "6275a64e-ec9f-408a-a234-37abfae3ec71" (UID: "6275a64e-ec9f-408a-a234-37abfae3ec71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:24:18.265306 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.265212 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-home" pod "6275a64e-ec9f-408a-a234-37abfae3ec71" (UID: "6275a64e-ec9f-408a-a234-37abfae3ec71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:24:18.265476 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.265360 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-model-cache" pod "6275a64e-ec9f-408a-a234-37abfae3ec71" (UID: "6275a64e-ec9f-408a-a234-37abfae3ec71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:24:18.265476 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.265464 4855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-model-cache\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:24:18.265629 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.265481 4855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-tmp-dir\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:24:18.265629 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.265494 4855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-home\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:24:18.267275 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.267253 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6275a64e-ec9f-408a-a234-37abfae3ec71-kube-api-access-cjstm" pod "6275a64e-ec9f-408a-a234-37abfae3ec71" (UID: "6275a64e-ec9f-408a-a234-37abfae3ec71"). InnerVolumeSpecName "kube-api-access-cjstm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:24:18.267381 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.267275 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6275a64e-ec9f-408a-a234-37abfae3ec71-tls-certs" pod "6275a64e-ec9f-408a-a234-37abfae3ec71" (UID: "6275a64e-ec9f-408a-a234-37abfae3ec71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:24:18.267381 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.267291 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-dshm" pod "6275a64e-ec9f-408a-a234-37abfae3ec71" (UID: "6275a64e-ec9f-408a-a234-37abfae3ec71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:24:18.366558 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.366524 4855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6275a64e-ec9f-408a-a234-37abfae3ec71-tls-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:24:18.366558 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.366558 4855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6275a64e-ec9f-408a-a234-37abfae3ec71-dshm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:24:18.367083 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.366574 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjstm\" (UniqueName: \"kubernetes.io/projected/6275a64e-ec9f-408a-a234-37abfae3ec71-kube-api-access-cjstm\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:24:18.498471 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.497372 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx"] Nov 03 06:24:18.507410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:18.504221 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6d432a37-kserve-75654fbc7c-pdmhx"] Nov 03 06:24:20.332342 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:20.332304 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6275a64e-ec9f-408a-a234-37abfae3ec71" path="/var/lib/kubelet/pods/6275a64e-ec9f-408a-a234-37abfae3ec71/volumes" Nov 03 06:24:25.453153 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:25.453071 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:24:28.883751 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:24:28.883716 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:24:41.550330 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:41.550288 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:24:42.747591 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.747558 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-687bf79496-whw4r"] Nov 03 06:24:42.748087 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.748069 4855 cpu_manager.go:414] "containerMap: removing container" podUID="6275a64e-ec9f-408a-a234-37abfae3ec71" containerName="main" Nov 03 06:24:42.748139 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.748088 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6275a64e-ec9f-408a-a234-37abfae3ec71" containerName="main" Nov 03 06:24:42.748139 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.748119 4855 cpu_manager.go:414] "containerMap: removing container" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" Nov 03 06:24:42.748139 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.748125 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c952cfa1-3312-4605-8557-a8f6cc85dcda" containerName="istio-proxy" Nov 03 06:24:42.748290 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.748193 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="6275a64e-ec9f-408a-a234-37abfae3ec71" containerName="main" Nov 03 06:24:42.751646 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.751616 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.782866 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.782838 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-687bf79496-whw4r"] Nov 03 06:24:42.848628 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.848598 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bklkq\" (UniqueName: \"kubernetes.io/projected/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-kube-api-access-bklkq\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.848828 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.848635 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-openshift-service-ca\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.848828 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.848673 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-metrics-certs\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.848828 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.848798 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-maas-api-tls\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.949621 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.949585 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bklkq\" (UniqueName: \"kubernetes.io/projected/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-kube-api-access-bklkq\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.949621 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.949628 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-openshift-service-ca\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.949828 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.949670 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-metrics-certs\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.949828 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.949697 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-maas-api-tls\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.950504 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.950476 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-openshift-service-ca\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.952293 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.952266 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-metrics-certs\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.952450 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.952427 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-maas-api-tls\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:42.974162 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:42.974135 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bklkq\" (UniqueName: \"kubernetes.io/projected/d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61-kube-api-access-bklkq\") pod \"maas-api-687bf79496-whw4r\" (UID: \"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61\") " pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:43.066802 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:43.066768 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:43.371574 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:43.371547 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-687bf79496-whw4r"] Nov 03 06:24:43.373865 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:24:43.373838 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0173fb2_fe7d_4f45_b2dd_96f4e9bfef61.slice/crio-ca244850fae39a35294dd90b7e696067a27980ede66d9c7c5ab2773798d66752 WatchSource:0}: Error finding container ca244850fae39a35294dd90b7e696067a27980ede66d9c7c5ab2773798d66752: Status 404 returned error can't find the container with id ca244850fae39a35294dd90b7e696067a27980ede66d9c7c5ab2773798d66752 Nov 03 06:24:43.375138 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:43.375122 4855 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 06:24:44.304974 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:44.304941 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" event={"ID":"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61","Type":"ContainerStarted","Data":"96f8e0a833164a0a06ff94cfd29e1f80b04b7ceb173158fbfd7f1f9a9837514b"} Nov 03 06:24:44.305514 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:44.304997 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" event={"ID":"d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61","Type":"ContainerStarted","Data":"ca244850fae39a35294dd90b7e696067a27980ede66d9c7c5ab2773798d66752"} Nov 03 06:24:44.305514 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:44.305080 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:44.350849 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:44.350796 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" podStartSLOduration=1.9801330780000002 podStartE2EDuration="2.350782008s" totalImagesPullingTime="370.64893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:24:44.349324154 +0000 UTC m=+1772.157711422" watchObservedRunningTime="2026-11-03 06:24:44.350782008 +0000 UTC m=+1772.159169275" Nov 03 06:24:50.316161 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:50.316127 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-687bf79496-whw4r" Nov 03 06:24:53.930904 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:53.930862 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr"] Nov 03 06:24:53.934686 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:53.934662 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:53.966550 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:53.966522 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e41e4c4d-serving-cert\"" Nov 03 06:24:53.968521 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:53.968498 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr"] Nov 03 06:24:54.064692 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.064661 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggbt9\" (UniqueName: \"kubernetes.io/projected/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-kube-api-access-ggbt9\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.064884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.064714 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-maas-api-tls\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.064884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.064833 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-metrics-certs\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.064974 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.064935 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-openshift-service-ca\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.166063 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.166025 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggbt9\" (UniqueName: \"kubernetes.io/projected/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-kube-api-access-ggbt9\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.166256 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.166084 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-maas-api-tls\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.166329 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.166249 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-metrics-certs\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.166329 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.166312 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-openshift-service-ca\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.166961 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.166936 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-openshift-service-ca\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.168521 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.168494 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-maas-api-tls\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.168618 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.168527 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-metrics-certs\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.194590 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.194532 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggbt9\" (UniqueName: \"kubernetes.io/projected/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-kube-api-access-ggbt9\") pod \"maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.247825 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.247792 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:54.551279 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:54.551245 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr"] Nov 03 06:24:54.553610 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:24:54.553585 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcaec5b_be8e_4f3b_95c8_4caa2a561b85.slice/crio-4a2dca3a18fd8de8657430bba53e06336f414382ab0a73b3a0643f3b1ce96ee5 WatchSource:0}: Error finding container 4a2dca3a18fd8de8657430bba53e06336f414382ab0a73b3a0643f3b1ce96ee5: Status 404 returned error can't find the container with id 4a2dca3a18fd8de8657430bba53e06336f414382ab0a73b3a0643f3b1ce96ee5 Nov 03 06:24:55.390146 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:55.390112 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" event={"ID":"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85","Type":"ContainerStarted","Data":"857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570"} Nov 03 06:24:55.390596 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:55.390165 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" event={"ID":"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85","Type":"ContainerStarted","Data":"4a2dca3a18fd8de8657430bba53e06336f414382ab0a73b3a0643f3b1ce96ee5"} Nov 03 06:24:55.390596 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:55.390256 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:24:55.434679 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:55.434629 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" podStartSLOduration=2.050546708 podStartE2EDuration="2.434615463s" totalImagesPullingTime="384.068755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:24:55.434024693 +0000 UTC m=+1783.242411961" watchObservedRunningTime="2026-11-03 06:24:55.434615463 +0000 UTC m=+1783.243002709" Nov 03 06:24:57.636286 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:57.636243 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:24:57.688226 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:57.688194 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk"] Nov 03 06:24:57.688442 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:57.688412 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" podUID="061fb5a9-3b3a-44e8-90b7-ab7d51b28496" containerName="maas-api" containerID="cri-o://7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c" gracePeriod=30 Nov 03 06:24:57.946544 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:57.946446 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" podUID="061fb5a9-3b3a-44e8-90b7-ab7d51b28496" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.31:8443/health\": dial tcp 10.133.0.31:8443: connect: connection refused" Nov 03 06:24:58.892251 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:24:58.892211 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:24:58.936734 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:58.936706 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:24:59.025567 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.025486 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-kube-api-access-lfhd2\" (UniqueName: \"kubernetes.io/projected/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-kube-api-access-lfhd2\") pod \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " Nov 03 06:24:59.025715 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.025596 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-maas-api-tls\") pod \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " Nov 03 06:24:59.025715 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.025646 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-openshift-service-ca\") pod \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " Nov 03 06:24:59.025715 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.025674 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-metrics-certs\" (UniqueName: \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-metrics-certs\") pod \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\" (UID: \"061fb5a9-3b3a-44e8-90b7-ab7d51b28496\") " Nov 03 06:24:59.026040 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.026009 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-openshift-service-ca" pod "061fb5a9-3b3a-44e8-90b7-ab7d51b28496" (UID: "061fb5a9-3b3a-44e8-90b7-ab7d51b28496"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:24:59.027621 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.027592 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-maas-api-tls" pod "061fb5a9-3b3a-44e8-90b7-ab7d51b28496" (UID: "061fb5a9-3b3a-44e8-90b7-ab7d51b28496"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:24:59.028016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.027996 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-metrics-certs" pod "061fb5a9-3b3a-44e8-90b7-ab7d51b28496" (UID: "061fb5a9-3b3a-44e8-90b7-ab7d51b28496"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:24:59.028016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.028009 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-kube-api-access-lfhd2" pod "061fb5a9-3b3a-44e8-90b7-ab7d51b28496" (UID: "061fb5a9-3b3a-44e8-90b7-ab7d51b28496"). InnerVolumeSpecName "kube-api-access-lfhd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:24:59.126485 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.126455 4855 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-maas-api-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:24:59.126485 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.126482 4855 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-openshift-service-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:24:59.126689 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.126494 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:24:59.126689 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.126504 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhd2\" (UniqueName: \"kubernetes.io/projected/061fb5a9-3b3a-44e8-90b7-ab7d51b28496-kube-api-access-lfhd2\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:24:59.410061 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.410023 4855 generic.go:356] "Generic (PLEG): container finished" podID="061fb5a9-3b3a-44e8-90b7-ab7d51b28496" containerID="7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c" exitCode=0 Nov 03 06:24:59.410271 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.410072 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" Nov 03 06:24:59.410271 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.410088 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" event={"ID":"061fb5a9-3b3a-44e8-90b7-ab7d51b28496","Type":"ContainerDied","Data":"7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c"} Nov 03 06:24:59.410271 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.410140 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk" event={"ID":"061fb5a9-3b3a-44e8-90b7-ab7d51b28496","Type":"ContainerDied","Data":"31f02660ef8e2e7adcb0c0671d36d5a9988df9ea22d74bdea97f350e0ffa8345"} Nov 03 06:24:59.410271 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.410168 4855 scope.go:122] "RemoveContainer" containerID="7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c" Nov 03 06:24:59.419818 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.419800 4855 scope.go:122] "RemoveContainer" containerID="7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c" Nov 03 06:24:59.420095 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:24:59.420077 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c\": container with ID starting with 7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c not found: ID does not exist" containerID="7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c" Nov 03 06:24:59.420143 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.420105 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c"} err="failed to get container status \"7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c\": rpc error: code = NotFound desc = could not find container \"7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c\": container with ID starting with 7a14ca6d9f448d0d197a8ec55cf754cc51f73acd3da4c14cd04c847f370c6f6c not found: ID does not exist" Nov 03 06:24:59.466096 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.466064 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk"] Nov 03 06:24:59.470736 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:24:59.470709 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fd5d8948d-sgjnk"] Nov 03 06:25:00.331662 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:00.331630 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061fb5a9-3b3a-44e8-90b7-ab7d51b28496" path="/var/lib/kubelet/pods/061fb5a9-3b3a-44e8-90b7-ab7d51b28496/volumes" Nov 03 06:25:01.401165 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:01.401135 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:25:12.545183 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:12.545155 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:25:12.549848 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:12.549823 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:25:21.482910 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.482875 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr"] Nov 03 06:25:21.483684 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.483238 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" podUID="6fcaec5b-be8e-4f3b-95c8-4caa2a561b85" containerName="maas-api" containerID="cri-o://857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570" gracePeriod=30 Nov 03 06:25:21.769947 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.769921 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:25:21.838098 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.838061 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-kube-api-access-ggbt9\" (UniqueName: \"kubernetes.io/projected/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-kube-api-access-ggbt9\") pod \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " Nov 03 06:25:21.838256 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.838201 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-maas-api-tls\") pod \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " Nov 03 06:25:21.838299 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.838277 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-metrics-certs\") pod \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " Nov 03 06:25:21.838344 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.838317 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-openshift-service-ca\") pod \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\" (UID: \"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85\") " Nov 03 06:25:21.838758 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.838726 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-openshift-service-ca" pod "6fcaec5b-be8e-4f3b-95c8-4caa2a561b85" (UID: "6fcaec5b-be8e-4f3b-95c8-4caa2a561b85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:25:21.840464 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.840434 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-maas-api-tls" pod "6fcaec5b-be8e-4f3b-95c8-4caa2a561b85" (UID: "6fcaec5b-be8e-4f3b-95c8-4caa2a561b85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:25:21.840688 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.840670 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-kube-api-access-ggbt9" pod "6fcaec5b-be8e-4f3b-95c8-4caa2a561b85" (UID: "6fcaec5b-be8e-4f3b-95c8-4caa2a561b85"). InnerVolumeSpecName "kube-api-access-ggbt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:25:21.840779 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.840680 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-metrics-certs" pod "6fcaec5b-be8e-4f3b-95c8-4caa2a561b85" (UID: "6fcaec5b-be8e-4f3b-95c8-4caa2a561b85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:25:21.939852 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.939815 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:25:21.939852 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.939853 4855 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-openshift-service-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:25:21.940084 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.939866 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggbt9\" (UniqueName: \"kubernetes.io/projected/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-kube-api-access-ggbt9\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:25:21.940084 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:21.939878 4855 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85-maas-api-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:25:22.520258 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:22.520227 4855 generic.go:356] "Generic (PLEG): container finished" podID="6fcaec5b-be8e-4f3b-95c8-4caa2a561b85" containerID="857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570" exitCode=0 Nov 03 06:25:22.520738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:22.520283 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" Nov 03 06:25:22.520738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:22.520314 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" event={"ID":"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85","Type":"ContainerDied","Data":"857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570"} Nov 03 06:25:22.520738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:22.520354 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr" event={"ID":"6fcaec5b-be8e-4f3b-95c8-4caa2a561b85","Type":"ContainerDied","Data":"4a2dca3a18fd8de8657430bba53e06336f414382ab0a73b3a0643f3b1ce96ee5"} Nov 03 06:25:22.520738 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:22.520374 4855 scope.go:122] "RemoveContainer" containerID="857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570" Nov 03 06:25:22.528926 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:22.528906 4855 scope.go:122] "RemoveContainer" containerID="857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570" Nov 03 06:25:22.529220 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:25:22.529202 4855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570\": container with ID starting with 857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570 not found: ID does not exist" containerID="857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570" Nov 03 06:25:22.529269 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:22.529230 4855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570"} err="failed to get container status \"857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570\": rpc error: code = NotFound desc = could not find container \"857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570\": container with ID starting with 857e6b6dcbbbadff40950ca8d8fab0f183e0512413c5375f4d659da7295b1570 not found: ID does not exist" Nov 03 06:25:22.574299 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:22.574272 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr"] Nov 03 06:25:22.580228 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:22.580208 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e41e4c4d-59964b47cf-td7qr"] Nov 03 06:25:23.580358 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:23.580315 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:25:24.332648 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:24.332613 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fcaec5b-be8e-4f3b-95c8-4caa2a561b85" path="/var/lib/kubelet/pods/6fcaec5b-be8e-4f3b-95c8-4caa2a561b85/volumes" Nov 03 06:25:26.335660 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.335624 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4"] Nov 03 06:25:26.336160 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.336127 4855 cpu_manager.go:414] "containerMap: removing container" podUID="061fb5a9-3b3a-44e8-90b7-ab7d51b28496" containerName="maas-api" Nov 03 06:25:26.336238 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.336162 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061fb5a9-3b3a-44e8-90b7-ab7d51b28496" containerName="maas-api" Nov 03 06:25:26.336238 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.336203 4855 cpu_manager.go:414] "containerMap: removing container" podUID="6fcaec5b-be8e-4f3b-95c8-4caa2a561b85" containerName="maas-api" Nov 03 06:25:26.336238 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.336212 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcaec5b-be8e-4f3b-95c8-4caa2a561b85" containerName="maas-api" Nov 03 06:25:26.336436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.336324 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcaec5b-be8e-4f3b-95c8-4caa2a561b85" containerName="maas-api" Nov 03 06:25:26.336436 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.336339 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="061fb5a9-3b3a-44e8-90b7-ab7d51b28496" containerName="maas-api" Nov 03 06:25:26.341944 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.341922 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.364026 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.364000 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-5797b0-openshift-default-dockercfg-qfz5m\"" Nov 03 06:25:26.379450 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.379418 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4"] Nov 03 06:25:26.387435 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.387405 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-socket\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.387572 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.387445 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-credential-socket\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.387712 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.387685 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-certs\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.387807 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.387786 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-podinfo\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.387856 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.387820 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef6a97e-255d-4952-8e36-f92e499dfd12-istiod-ca-cert\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.387856 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.387843 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-data\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.387960 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.387865 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-token\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.387960 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.387924 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-envoy\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.388073 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.388047 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5n8\" (UniqueName: \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-kube-api-access-rl5n8\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.489762 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.489714 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-podinfo\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.490095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.489777 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef6a97e-255d-4952-8e36-f92e499dfd12-istiod-ca-cert\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.490095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.489808 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-data\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.490095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.489839 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-token\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.490095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.489874 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-envoy\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.490095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.489912 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5n8\" (UniqueName: \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-kube-api-access-rl5n8\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.490095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.489955 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-socket\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.490095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.489981 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-credential-socket\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.490558 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.490112 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-certs\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.490662 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.490600 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-certs\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.491587 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.491561 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-socket\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.491697 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.491669 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-credential-socket\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.491774 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.491759 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef6a97e-255d-4952-8e36-f92e499dfd12-istiod-ca-cert\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.491962 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.491934 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-data\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.493645 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.493617 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-podinfo\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.493785 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.493768 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-envoy\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.516863 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.516830 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5n8\" (UniqueName: \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-kube-api-access-rl5n8\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.518963 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.518927 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-token\") pod \"e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.657531 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.657449 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:26.975060 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.975033 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4"] Nov 03 06:25:26.977413 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:25:26.977360 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef6a97e_255d_4952_8e36_f92e499dfd12.slice/crio-8adfb4f03681d57060fa6bcca8d3ece3632be74acc52f0a51df91e70a32ed4be WatchSource:0}: Error finding container 8adfb4f03681d57060fa6bcca8d3ece3632be74acc52f0a51df91e70a32ed4be: Status 404 returned error can't find the container with id 8adfb4f03681d57060fa6bcca8d3ece3632be74acc52f0a51df91e70a32ed4be Nov 03 06:25:26.979664 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.979629 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:25:26.979747 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.979696 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:25:26.979747 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:26.979723 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:25:27.547784 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:27.547751 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" event={"ID":"5ef6a97e-255d-4952-8e36-f92e499dfd12","Type":"ContainerStarted","Data":"c1bfbb7898ce92ea4a472f641724d0201da773c991dc7055e6a6bc0427d1cb81"} Nov 03 06:25:27.548182 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:27.547805 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" event={"ID":"5ef6a97e-255d-4952-8e36-f92e499dfd12","Type":"ContainerStarted","Data":"8adfb4f03681d57060fa6bcca8d3ece3632be74acc52f0a51df91e70a32ed4be"} Nov 03 06:25:27.610947 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:27.610895 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" podStartSLOduration=1.610881223 podStartE2EDuration="1.610881223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:25:27.609180264 +0000 UTC m=+1815.417567531" watchObservedRunningTime="2026-11-03 06:25:27.610881223 +0000 UTC m=+1815.419268469" Nov 03 06:25:27.658553 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:27.658521 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:28.663345 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:28.663316 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:28.884054 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:25:28.884016 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:25:29.558201 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:29.558168 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:29.559298 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:29.559273 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:31.565613 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.565581 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t"] Nov 03 06:25:31.569526 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.569505 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.585487 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.585467 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-5797b0-serving-cert\"" Nov 03 06:25:31.602424 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.601468 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t"] Nov 03 06:25:31.646894 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.646861 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.646894 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.646894 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7wch\" (UniqueName: \"kubernetes.io/projected/d3194566-0d7e-4a86-ad54-502a5756666b-kube-api-access-q7wch\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.647094 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.647052 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3194566-0d7e-4a86-ad54-502a5756666b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.647094 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.647085 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.748508 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.748477 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3194566-0d7e-4a86-ad54-502a5756666b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.748691 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.748517 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.748691 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.748587 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.748691 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.748609 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7wch\" (UniqueName: \"kubernetes.io/projected/d3194566-0d7e-4a86-ad54-502a5756666b-kube-api-access-q7wch\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.749193 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.749165 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3194566-0d7e-4a86-ad54-502a5756666b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.751117 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.751093 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.751222 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.751118 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.773129 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.773107 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7wch\" (UniqueName: \"kubernetes.io/projected/d3194566-0d7e-4a86-ad54-502a5756666b-kube-api-access-q7wch\") pod \"maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:31.882374 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:31.882291 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:32.183745 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:32.183716 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t"] Nov 03 06:25:32.185955 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:25:32.185927 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3194566_0d7e_4a86_ad54_502a5756666b.slice/crio-32b5249ff1de679801f217e76d3cc62c47bb2c03641428fd3ff414530474534e WatchSource:0}: Error finding container 32b5249ff1de679801f217e76d3cc62c47bb2c03641428fd3ff414530474534e: Status 404 returned error can't find the container with id 32b5249ff1de679801f217e76d3cc62c47bb2c03641428fd3ff414530474534e Nov 03 06:25:32.573128 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:32.573102 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" event={"ID":"d3194566-0d7e-4a86-ad54-502a5756666b","Type":"ContainerStarted","Data":"32b5249ff1de679801f217e76d3cc62c47bb2c03641428fd3ff414530474534e"} Nov 03 06:25:33.579449 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:33.579414 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" event={"ID":"d3194566-0d7e-4a86-ad54-502a5756666b","Type":"ContainerStarted","Data":"17afb7cc5d240d401386eb3d444cb71e7fc5ffdc0aeb177d921dfdb7acee31e1"} Nov 03 06:25:33.579904 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:33.579522 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:33.624520 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:33.624452 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" podStartSLOduration=2.268880046 podStartE2EDuration="2.624433393s" totalImagesPullingTime="355.553347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:25:33.624129334 +0000 UTC m=+1821.432516603" watchObservedRunningTime="2026-11-03 06:25:33.624433393 +0000 UTC m=+1821.432820663" Nov 03 06:25:35.428589 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:35.428555 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:25:39.590003 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:39.589971 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:25:44.559410 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:44.559359 4855 patch_prober.go:29] interesting pod/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 06:25:44.559821 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:44.559451 4855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 06:25:44.632996 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:44.632923 4855 generic.go:356] "Generic (PLEG): container finished" podID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerID="c1bfbb7898ce92ea4a472f641724d0201da773c991dc7055e6a6bc0427d1cb81" exitCode=0 Nov 03 06:25:44.633137 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:44.632998 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" event={"ID":"5ef6a97e-255d-4952-8e36-f92e499dfd12","Type":"ContainerDied","Data":"c1bfbb7898ce92ea4a472f641724d0201da773c991dc7055e6a6bc0427d1cb81"} Nov 03 06:25:44.633361 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:44.633347 4855 scope.go:122] "RemoveContainer" containerID="c1bfbb7898ce92ea4a472f641724d0201da773c991dc7055e6a6bc0427d1cb81" Nov 03 06:25:44.634484 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:44.634457 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:25:44.634592 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:44.634538 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:25:44.634592 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:44.634585 4855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 06:25:45.659945 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:45.659909 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" event={"ID":"5ef6a97e-255d-4952-8e36-f92e499dfd12","Type":"ContainerStarted","Data":"abc7b0c0803bb05da4e81a94d1e7c49397b1dcdb8f60b1e5fb29e7314adbb6e5"} Nov 03 06:25:46.664033 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:46.663998 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:46.664747 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:46.664720 4855 patch_prober.go:29] interesting pod/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 06:25:46.664942 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:46.664915 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 06:25:47.658893 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:47.658852 4855 patch_prober.go:29] interesting pod/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 06:25:47.659078 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:47.658928 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 06:25:48.658081 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:48.658042 4855 patch_prober.go:29] interesting pod/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 06:25:48.658669 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:48.658117 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 06:25:49.657931 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:49.657886 4855 patch_prober.go:29] interesting pod/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 06:25:49.658155 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:49.657999 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 06:25:50.658856 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:50.658813 4855 patch_prober.go:29] interesting pod/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 06:25:50.659261 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:50.658882 4855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 06:25:51.663099 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:51.663071 4855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:51.663505 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:51.663411 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:51.664012 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:25:51.663992 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:25:58.883165 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:25:58.883074 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:26:28.882952 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:26:28.882915 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:26:58.884741 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:26:58.884703 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:27:21.972082 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:21.972046 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t"] Nov 03 06:27:21.972647 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:21.972286 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" podUID="d3194566-0d7e-4a86-ad54-502a5756666b" containerName="maas-api" containerID="cri-o://17afb7cc5d240d401386eb3d444cb71e7fc5ffdc0aeb177d921dfdb7acee31e1" gracePeriod=30 Nov 03 06:27:22.130451 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.130411 4855 generic.go:356] "Generic (PLEG): container finished" podID="d3194566-0d7e-4a86-ad54-502a5756666b" containerID="17afb7cc5d240d401386eb3d444cb71e7fc5ffdc0aeb177d921dfdb7acee31e1" exitCode=0 Nov 03 06:27:22.130616 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.130473 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" event={"ID":"d3194566-0d7e-4a86-ad54-502a5756666b","Type":"ContainerDied","Data":"17afb7cc5d240d401386eb3d444cb71e7fc5ffdc0aeb177d921dfdb7acee31e1"} Nov 03 06:27:22.262193 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.262118 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:27:22.348009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.347981 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-maas-api-tls\") pod \"d3194566-0d7e-4a86-ad54-502a5756666b\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " Nov 03 06:27:22.348207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.348021 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-metrics-certs\") pod \"d3194566-0d7e-4a86-ad54-502a5756666b\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " Nov 03 06:27:22.348207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.348052 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3194566-0d7e-4a86-ad54-502a5756666b-kube-api-access-q7wch\" (UniqueName: \"kubernetes.io/projected/d3194566-0d7e-4a86-ad54-502a5756666b-kube-api-access-q7wch\") pod \"d3194566-0d7e-4a86-ad54-502a5756666b\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " Nov 03 06:27:22.348207 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.348071 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3194566-0d7e-4a86-ad54-502a5756666b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3194566-0d7e-4a86-ad54-502a5756666b-openshift-service-ca\") pod \"d3194566-0d7e-4a86-ad54-502a5756666b\" (UID: \"d3194566-0d7e-4a86-ad54-502a5756666b\") " Nov 03 06:27:22.348513 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.348491 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3194566-0d7e-4a86-ad54-502a5756666b-openshift-service-ca" pod "d3194566-0d7e-4a86-ad54-502a5756666b" (UID: "d3194566-0d7e-4a86-ad54-502a5756666b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:27:22.350225 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.350196 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-maas-api-tls" pod "d3194566-0d7e-4a86-ad54-502a5756666b" (UID: "d3194566-0d7e-4a86-ad54-502a5756666b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:27:22.350322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.350220 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3194566-0d7e-4a86-ad54-502a5756666b-kube-api-access-q7wch" pod "d3194566-0d7e-4a86-ad54-502a5756666b" (UID: "d3194566-0d7e-4a86-ad54-502a5756666b"). InnerVolumeSpecName "kube-api-access-q7wch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:27:22.350322 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.350248 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-metrics-certs" pod "d3194566-0d7e-4a86-ad54-502a5756666b" (UID: "d3194566-0d7e-4a86-ad54-502a5756666b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:27:22.453769 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.453731 4855 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-maas-api-tls\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:22.453769 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.453763 4855 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3194566-0d7e-4a86-ad54-502a5756666b-metrics-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:22.453769 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.453775 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7wch\" (UniqueName: \"kubernetes.io/projected/d3194566-0d7e-4a86-ad54-502a5756666b-kube-api-access-q7wch\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:22.454027 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:22.453788 4855 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3194566-0d7e-4a86-ad54-502a5756666b-openshift-service-ca\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:23.136122 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:23.136089 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" event={"ID":"d3194566-0d7e-4a86-ad54-502a5756666b","Type":"ContainerDied","Data":"32b5249ff1de679801f217e76d3cc62c47bb2c03641428fd3ff414530474534e"} Nov 03 06:27:23.136652 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:23.136130 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t" Nov 03 06:27:23.136652 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:23.136149 4855 scope.go:122] "RemoveContainer" containerID="17afb7cc5d240d401386eb3d444cb71e7fc5ffdc0aeb177d921dfdb7acee31e1" Nov 03 06:27:23.192588 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:23.192558 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t"] Nov 03 06:27:23.205585 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:23.205560 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5797b0-9d54bbb78-27f5t"] Nov 03 06:27:24.236027 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:24.235111 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:27:24.331996 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:24.331957 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3194566-0d7e-4a86-ad54-502a5756666b" path="/var/lib/kubelet/pods/d3194566-0d7e-4a86-ad54-502a5756666b/volumes" Nov 03 06:27:28.883012 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:27:28.882967 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:27:29.838564 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:29.838533 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4"] Nov 03 06:27:29.838824 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:29.838795 4855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" containerID="cri-o://abc7b0c0803bb05da4e81a94d1e7c49397b1dcdb8f60b1e5fb29e7314adbb6e5" gracePeriod=30 Nov 03 06:27:32.340666 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:32.340635 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:27:35.216040 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.213652 4855 generic.go:356] "Generic (PLEG): container finished" podID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerID="abc7b0c0803bb05da4e81a94d1e7c49397b1dcdb8f60b1e5fb29e7314adbb6e5" exitCode=0 Nov 03 06:27:35.216040 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.213718 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" event={"ID":"5ef6a97e-255d-4952-8e36-f92e499dfd12","Type":"ContainerDied","Data":"abc7b0c0803bb05da4e81a94d1e7c49397b1dcdb8f60b1e5fb29e7314adbb6e5"} Nov 03 06:27:35.216040 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.213793 4855 scope.go:122] "RemoveContainer" containerID="c1bfbb7898ce92ea4a472f641724d0201da773c991dc7055e6a6bc0427d1cb81" Nov 03 06:27:35.243609 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.243579 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:27:35.413208 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413174 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-certs\") pod \"5ef6a97e-255d-4952-8e36-f92e499dfd12\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " Nov 03 06:27:35.413418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413240 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-credential-socket\") pod \"5ef6a97e-255d-4952-8e36-f92e499dfd12\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " Nov 03 06:27:35.413418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413278 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-envoy\") pod \"5ef6a97e-255d-4952-8e36-f92e499dfd12\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " Nov 03 06:27:35.413418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413347 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-token\" (UniqueName: \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-token\") pod \"5ef6a97e-255d-4952-8e36-f92e499dfd12\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " Nov 03 06:27:35.413418 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413381 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef6a97e-255d-4952-8e36-f92e499dfd12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef6a97e-255d-4952-8e36-f92e499dfd12-istiod-ca-cert\") pod \"5ef6a97e-255d-4952-8e36-f92e499dfd12\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " Nov 03 06:27:35.413658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413441 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-kube-api-access-rl5n8\" (UniqueName: \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-kube-api-access-rl5n8\") pod \"5ef6a97e-255d-4952-8e36-f92e499dfd12\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " Nov 03 06:27:35.413658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413475 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-data\") pod \"5ef6a97e-255d-4952-8e36-f92e499dfd12\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " Nov 03 06:27:35.413658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413536 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-certs" pod "5ef6a97e-255d-4952-8e36-f92e499dfd12" (UID: "5ef6a97e-255d-4952-8e36-f92e499dfd12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:27:35.413658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413555 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-podinfo\") pod \"5ef6a97e-255d-4952-8e36-f92e499dfd12\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " Nov 03 06:27:35.413658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413616 4855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-socket\") pod \"5ef6a97e-255d-4952-8e36-f92e499dfd12\" (UID: \"5ef6a97e-255d-4952-8e36-f92e499dfd12\") " Nov 03 06:27:35.413658 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413637 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-credential-socket" pod "5ef6a97e-255d-4952-8e36-f92e499dfd12" (UID: "5ef6a97e-255d-4952-8e36-f92e499dfd12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:27:35.413973 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413785 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-data" pod "5ef6a97e-255d-4952-8e36-f92e499dfd12" (UID: "5ef6a97e-255d-4952-8e36-f92e499dfd12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:27:35.413973 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413886 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef6a97e-255d-4952-8e36-f92e499dfd12-istiod-ca-cert" pod "5ef6a97e-255d-4952-8e36-f92e499dfd12" (UID: "5ef6a97e-255d-4952-8e36-f92e499dfd12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:27:35.413973 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.413956 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-socket" pod "5ef6a97e-255d-4952-8e36-f92e499dfd12" (UID: "5ef6a97e-255d-4952-8e36-f92e499dfd12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:27:35.414106 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.414092 4855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef6a97e-255d-4952-8e36-f92e499dfd12-istiod-ca-cert\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:35.414150 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.414111 4855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-data\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:35.414150 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.414124 4855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:35.414150 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.414136 4855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-workload-certs\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:35.414272 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.414150 4855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-credential-socket\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:35.415792 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.415763 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-token" pod "5ef6a97e-255d-4952-8e36-f92e499dfd12" (UID: "5ef6a97e-255d-4952-8e36-f92e499dfd12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:27:35.415898 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.415790 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-podinfo" pod "5ef6a97e-255d-4952-8e36-f92e499dfd12" (UID: "5ef6a97e-255d-4952-8e36-f92e499dfd12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 06:27:35.416056 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.416022 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-kube-api-access-rl5n8" pod "5ef6a97e-255d-4952-8e36-f92e499dfd12" (UID: "5ef6a97e-255d-4952-8e36-f92e499dfd12"). InnerVolumeSpecName "kube-api-access-rl5n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:27:35.416056 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.416036 4855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-envoy" pod "5ef6a97e-255d-4952-8e36-f92e499dfd12" (UID: "5ef6a97e-255d-4952-8e36-f92e499dfd12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:27:35.515703 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.515619 4855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-envoy\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:35.515703 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.515647 4855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-token\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:35.515703 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.515657 4855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5n8\" (UniqueName: \"kubernetes.io/projected/5ef6a97e-255d-4952-8e36-f92e499dfd12-kube-api-access-rl5n8\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:35.515703 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:35.515667 4855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef6a97e-255d-4952-8e36-f92e499dfd12-istio-podinfo\") on node \"ip-10-0-141-67.ec2.internal\" DevicePath \"\"" Nov 03 06:27:36.219990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:36.219958 4855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" Nov 03 06:27:36.219990 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:36.219970 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4" event={"ID":"5ef6a97e-255d-4952-8e36-f92e499dfd12","Type":"ContainerDied","Data":"8adfb4f03681d57060fa6bcca8d3ece3632be74acc52f0a51df91e70a32ed4be"} Nov 03 06:27:36.220543 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:36.220064 4855 scope.go:122] "RemoveContainer" containerID="abc7b0c0803bb05da4e81a94d1e7c49397b1dcdb8f60b1e5fb29e7314adbb6e5" Nov 03 06:27:36.320884 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:36.320840 4855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4"] Nov 03 06:27:36.325333 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:36.325301 4855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5797b0-openshift-default-7888b776c7-g4zf4"] Nov 03 06:27:36.331756 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:36.331730 4855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" path="/var/lib/kubelet/pods/5ef6a97e-255d-4952-8e36-f92e499dfd12/volumes" Nov 03 06:27:40.767618 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:40.767549 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:27:58.633586 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:27:58.633546 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:27:58.883897 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:27:58.883813 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:28:26.729674 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:26.729641 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qfq2f"] Nov 03 06:28:28.887559 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:28:28.887512 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:28:38.398038 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:38.398002 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-678d788c65-zrb8q_549a963a-6ae5-48c8-a7f2-2935d181e004/manager/0.log" Nov 03 06:28:39.100016 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:39.099987 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gth2s_e5babcd3-d9e7-409d-a677-fa56d8d024c2/proxy/0.log" Nov 03 06:28:40.018046 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:40.018015 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-687bf79496-whw4r_d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61/maas-api/0.log" Nov 03 06:28:43.788314 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:43.788280 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qfq2f_0270da45-d870-4752-aef3-85f92cebb57e/limitador/0.log" Nov 03 06:28:44.935130 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:44.935097 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r2pjg_5cae47d3-8f00-409f-8a46-6d982382e33b/discovery/0.log" Nov 03 06:28:45.142707 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:45.142674 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d95cd9746-lvxsh_f7fa46c9-0a19-4980-b190-06d98b2aaf8e/kube-auth-proxy/0.log" Nov 03 06:28:46.465799 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:46.465771 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nkx8k_fd2713fe-2336-4f1e-8013-f8ede390211d/istio-proxy/0.log" Nov 03 06:28:46.668760 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:46.668733 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nnlxj_91014878-7b02-4ef8-8921-5bcef4cefa7b/istio-proxy/0.log" Nov 03 06:28:47.060376 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:47.060348 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ssx5p_17697121-d871-43b4-b1e8-5e74fb513cd5/istio-proxy/0.log" Nov 03 06:28:47.991727 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:47.991701 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75b97bb597-swr58_76c09b47-0ab3-4212-924b-131d88e94ab3/router/0.log" Nov 03 06:28:49.179382 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:49.179351 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-dhcm9_ee25c503-9d87-47a1-b630-e0d6d24e1c18/main/0.log" Nov 03 06:28:49.730865 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:49.730836 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2_5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d/main/0.log" Nov 03 06:28:49.926062 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:49.926037 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ds4ns_a65e242d-508a-4b6e-860f-f9095ccb686b/main/0.log" Nov 03 06:28:50.285597 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:28:50.285568 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5_ec588a2d-3f62-4256-afed-c86041c5a59b/main/0.log" Nov 03 06:28:58.882773 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:28:58.882678 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:29:01.060432 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:01.060383 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-678d788c65-zrb8q_549a963a-6ae5-48c8-a7f2-2935d181e004/manager/0.log" Nov 03 06:29:01.715230 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:01.715204 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gth2s_e5babcd3-d9e7-409d-a677-fa56d8d024c2/proxy/0.log" Nov 03 06:29:02.537251 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:02.537222 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-687bf79496-whw4r_d0173fb2-fe7d-4f45-b2dd-96f4e9bfef61/maas-api/0.log" Nov 03 06:29:05.880352 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:05.880324 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qfq2f_0270da45-d870-4752-aef3-85f92cebb57e/limitador/0.log" Nov 03 06:29:06.852936 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:06.852903 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r2pjg_5cae47d3-8f00-409f-8a46-6d982382e33b/discovery/0.log" Nov 03 06:29:07.075841 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:07.075802 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d95cd9746-lvxsh_f7fa46c9-0a19-4980-b190-06d98b2aaf8e/kube-auth-proxy/0.log" Nov 03 06:29:08.469943 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:08.469916 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nkx8k_fd2713fe-2336-4f1e-8013-f8ede390211d/istio-proxy/0.log" Nov 03 06:29:08.668822 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:08.668786 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nnlxj_91014878-7b02-4ef8-8921-5bcef4cefa7b/istio-proxy/0.log" Nov 03 06:29:09.061364 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:09.061336 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ssx5p_17697121-d871-43b4-b1e8-5e74fb513cd5/istio-proxy/0.log" Nov 03 06:29:10.026794 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:10.026766 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75b97bb597-swr58_76c09b47-0ab3-4212-924b-131d88e94ab3/router/0.log" Nov 03 06:29:10.734838 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:10.734809 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-dhcm9_ee25c503-9d87-47a1-b630-e0d6d24e1c18/main/0.log" Nov 03 06:29:11.310874 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:11.310834 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbln2_5ed1974f-ec09-4cd4-b5e9-5e7211f13b9d/main/0.log" Nov 03 06:29:11.514546 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:11.514515 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ds4ns_a65e242d-508a-4b6e-860f-f9095ccb686b/main/0.log" Nov 03 06:29:11.894868 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:11.894836 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-l7xj5_ec588a2d-3f62-4256-afed-c86041c5a59b/main/0.log" Nov 03 06:29:28.883142 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:29:28.883107 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:29:48.439347 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:48.439317 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-678d788c65-zrb8q_549a963a-6ae5-48c8-a7f2-2935d181e004/manager/0.log" Nov 03 06:29:48.721581 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:29:48.721498 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-678d788c65-zrb8q_549a963a-6ae5-48c8-a7f2-2935d181e004/manager/0.log" Nov 03 06:29:58.882876 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:29:58.882832 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:30:12.583143 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:12.583113 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:30:12.586270 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:12.586248 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:30:28.883021 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:30:28.882924 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:30:35.515385 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:35.515352 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-678d788c65-zrb8q_549a963a-6ae5-48c8-a7f2-2935d181e004/manager/0.log" Nov 03 06:30:35.809603 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:35.809569 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-678d788c65-zrb8q_549a963a-6ae5-48c8-a7f2-2935d181e004/manager/0.log" Nov 03 06:30:41.610102 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:41.610069 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vspq2_d73ab4e1-b560-420b-8216-6272bd184f39/global-pull-secret-syncer/0.log" Nov 03 06:30:41.781111 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:41.781079 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6z5gg_9ef729d9-2e44-4c39-a018-5df7a941dcc5/connection-checker/0.log" Nov 03 06:30:41.896982 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:41.896912 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bhkn5_db236f0a-264d-4498-8dd1-1787e2a3ee95/connection-checker/0.log" Nov 03 06:30:42.008979 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:42.008953 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tnzjq_054b9d40-fa74-4f73-9db4-70ff0dd43191/connection-checker/0.log" Nov 03 06:30:42.247763 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:42.247683 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sp2qh_7502a388-bbcd-4916-8b8d-a4258c58d575/konnectivity-agent/0.log" Nov 03 06:30:42.766565 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:42.766534 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-67.ec2.internal_45ef46b1ad95006153c299e1a6c8cea2/haproxy/0.log" Nov 03 06:30:49.572222 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:49.572192 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qfq2f_0270da45-d870-4752-aef3-85f92cebb57e/limitador/0.log" Nov 03 06:30:51.137469 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.137430 4855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt"] Nov 03 06:30:51.137882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.137837 4855 cpu_manager.go:414] "containerMap: removing container" podUID="d3194566-0d7e-4a86-ad54-502a5756666b" containerName="maas-api" Nov 03 06:30:51.137882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.137850 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3194566-0d7e-4a86-ad54-502a5756666b" containerName="maas-api" Nov 03 06:30:51.137882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.137872 4855 cpu_manager.go:414] "containerMap: removing container" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" Nov 03 06:30:51.137882 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.137879 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" Nov 03 06:30:51.138020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.137888 4855 cpu_manager.go:414] "containerMap: removing container" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" Nov 03 06:30:51.138020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.137893 4855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" Nov 03 06:30:51.138020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.137959 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" Nov 03 06:30:51.138020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.137965 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef6a97e-255d-4952-8e36-f92e499dfd12" containerName="istio-proxy" Nov 03 06:30:51.138020 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.137974 4855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3194566-0d7e-4a86-ad54-502a5756666b" containerName="maas-api" Nov 03 06:30:51.140936 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.140915 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.157904 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.157876 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c9b79\"/\"openshift-service-ca.crt\"" Nov 03 06:30:51.158095 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.157883 4855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c9b79\"/\"default-dockercfg-b9xm5\"" Nov 03 06:30:51.171345 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.171320 4855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c9b79\"/\"kube-root-ca.crt\"" Nov 03 06:30:51.173137 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.173115 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt"] Nov 03 06:30:51.209736 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.209692 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-sys\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.209913 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.209752 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-lib-modules\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.209913 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.209817 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-podres\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.209998 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.209911 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-proc\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.209998 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.209961 4855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mchpv\" (UniqueName: \"kubernetes.io/projected/60f1057a-4c4e-432c-a465-b582810ca9f2-kube-api-access-mchpv\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.311353 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.311320 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mchpv\" (UniqueName: \"kubernetes.io/projected/60f1057a-4c4e-432c-a465-b582810ca9f2-kube-api-access-mchpv\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.311583 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.311368 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-sys\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.311583 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.311412 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-lib-modules\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.311583 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.311437 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-podres\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.311583 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.311476 4855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-proc\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.311583 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.311495 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-sys\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.311583 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.311516 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-lib-modules\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.311583 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.311563 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-proc\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.311821 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.311579 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60f1057a-4c4e-432c-a465-b582810ca9f2-podres\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.335716 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.335686 4855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mchpv\" (UniqueName: \"kubernetes.io/projected/60f1057a-4c4e-432c-a465-b582810ca9f2-kube-api-access-mchpv\") pod \"perf-node-gather-daemonset-c8zxt\" (UID: \"60f1057a-4c4e-432c-a465-b582810ca9f2\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.453980 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.453894 4855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:51.751735 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.751660 4855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt"] Nov 03 06:30:51.754126 ip-10-0-141-67 kubenswrapper[4855]: W1103 06:30:51.754098 4855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod60f1057a_4c4e_432c_a465_b582810ca9f2.slice/crio-93221a77f83e129ea5f44cf8f2035ba3c6df18cf77af8f8a050fe232309ad465 WatchSource:0}: Error finding container 93221a77f83e129ea5f44cf8f2035ba3c6df18cf77af8f8a050fe232309ad465: Status 404 returned error can't find the container with id 93221a77f83e129ea5f44cf8f2035ba3c6df18cf77af8f8a050fe232309ad465 Nov 03 06:30:51.854707 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:51.854679 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-vpktf_56b77a5b-a55e-4849-9789-b9c5a3db4b02/cluster-monitoring-operator/0.log" Nov 03 06:30:52.148902 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:52.148864 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" event={"ID":"60f1057a-4c4e-432c-a465-b582810ca9f2","Type":"ContainerStarted","Data":"cc6392605a4c5dd25033bc4b2302c13988103cb39eca257dae8805d19fd0dc77"} Nov 03 06:30:52.149327 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:52.148923 4855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" event={"ID":"60f1057a-4c4e-432c-a465-b582810ca9f2","Type":"ContainerStarted","Data":"93221a77f83e129ea5f44cf8f2035ba3c6df18cf77af8f8a050fe232309ad465"} Nov 03 06:30:52.149327 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:52.148945 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:52.192287 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:52.192232 4855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" podStartSLOduration=1.19221825 podStartE2EDuration="1.19221825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:30:52.190908992 +0000 UTC m=+2139.999296262" watchObservedRunningTime="2026-11-03 06:30:52.19221825 +0000 UTC m=+2140.000605518" Nov 03 06:30:52.547883 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:52.547853 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blr44_2eaab29a-3c3a-4eb1-ae3b-ba66066b1144/node-exporter/0.log" Nov 03 06:30:52.663935 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:52.663906 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blr44_2eaab29a-3c3a-4eb1-ae3b-ba66066b1144/kube-rbac-proxy/0.log" Nov 03 06:30:52.779590 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:52.779558 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blr44_2eaab29a-3c3a-4eb1-ae3b-ba66066b1144/init-textfile/0.log" Nov 03 06:30:57.290680 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:57.290649 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-zknxx_8ba1e446-7be2-4673-9a44-a6c6a5e1be2d/networking-console-plugin/0.log" Nov 03 06:30:58.163786 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:58.163760 4855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-c8zxt" Nov 03 06:30:58.337506 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:58.337476 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/2.log" Nov 03 06:30:58.340948 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:30:58.340926 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qwvxg_2b099592-879f-4313-be96-60d39b44f8a8/console-operator/3.log" Nov 03 06:30:58.883167 ip-10-0-141-67 kubenswrapper[4855]: E1103 06:30:58.883131 4855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqt2f" containerName="extractor" Nov 03 06:31:00.512913 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:00.512886 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-74hnk_42da3e8e-febb-4e95-ae4c-c750c4bd3481/volume-data-source-validator/0.log" Nov 03 06:31:03.170573 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:03.170548 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g9htv_bd3a09f2-9d41-40f6-94f1-6cb00971d950/dns/0.log" Nov 03 06:31:03.287154 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:03.287124 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g9htv_bd3a09f2-9d41-40f6-94f1-6cb00971d950/kube-rbac-proxy/0.log" Nov 03 06:31:03.628807 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:03.628775 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-99n4t_36beafc8-4aee-448b-8dfd-64f89637aa9f/dns-node-resolver/0.log" Nov 03 06:31:05.284944 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:05.284907 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vvkcc_1eb02066-c20a-45b3-885a-0e9834b1c998/node-ca/0.log" Nov 03 06:31:05.291901 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:05.291879 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r2pjg_5cae47d3-8f00-409f-8a46-6d982382e33b/discovery/0.log" Nov 03 06:31:05.325678 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:05.325652 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d95cd9746-lvxsh_f7fa46c9-0a19-4980-b190-06d98b2aaf8e/kube-auth-proxy/0.log" Nov 03 06:31:05.733094 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:05.733061 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nkx8k_fd2713fe-2336-4f1e-8013-f8ede390211d/istio-proxy/0.log" Nov 03 06:31:05.795032 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:05.795007 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nnlxj_91014878-7b02-4ef8-8921-5bcef4cefa7b/istio-proxy/0.log" Nov 03 06:31:05.916513 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:05.916483 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ssx5p_17697121-d871-43b4-b1e8-5e74fb513cd5/istio-proxy/0.log" Nov 03 06:31:06.211382 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:06.211353 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75b97bb597-swr58_76c09b47-0ab3-4212-924b-131d88e94ab3/router/0.log" Nov 03 06:31:08.263660 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:08.263627 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r2pjg_5cae47d3-8f00-409f-8a46-6d982382e33b/discovery/0.log" Nov 03 06:31:08.361009 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:08.360968 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d95cd9746-lvxsh_f7fa46c9-0a19-4980-b190-06d98b2aaf8e/kube-auth-proxy/0.log" Nov 03 06:31:09.149490 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:09.149458 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nkx8k_fd2713fe-2336-4f1e-8013-f8ede390211d/istio-proxy/0.log" Nov 03 06:31:09.297751 ip-10-0-141-67 kubenswrapper[4855]: I1103 06:31:09.297722 4855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-nnlxj_91014878-7b02-4ef8-8921-5bcef4cefa7b/istio-proxy/0.log"